Sep 16 18:39:21.546734 ip-10-0-130-204 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:39:21.952466 ip-10-0-130-204 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:39:21.952466 ip-10-0-130-204 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:39:21.952466 ip-10-0-130-204 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:39:21.952466 ip-10-0-130-204 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:39:21.952466 ip-10-0-130-204 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:39:21.953615 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.953516    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:39:21.957984 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.957961    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:21.957984 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.957980    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:21.957984 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.957985    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:21.957984 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.957991    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.957996    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958000    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958004    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958008    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958012    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958016    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958020    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958024    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958027    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958031    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958035    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958039    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958043    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958047    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958050    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958064    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958068    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958072    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958076    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:21.958241 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958080    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958084    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958088    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958092    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958101    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958105    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958109    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958113    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958117    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958123    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958130    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958134    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958140    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958145    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958150    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958155    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958159    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958165    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958169    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958174    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:21.959243 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958178    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:21.960107 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958183    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:21.960107 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958187    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:21.960107 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958192    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:21.960107 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958196    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:21.960107 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958200    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:21.960107 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958205    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:21.960107 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958209    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:21.960107 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958213    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:21.960107 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958218    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:21.960107 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958223    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:21.960107 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958227    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:21.960107 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958231    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:21.960107 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958236    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:21.960107 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958239    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:21.960107 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958243    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:21.960107 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958250    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:21.960107 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958255    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:21.960107 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958264    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:21.960107 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958270    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:21.960924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958276    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:21.960924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958281    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:21.960924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958285    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:21.960924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958289    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:21.960924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958295    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:21.960924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958299    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:21.960924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958304    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:21.960924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958308    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:21.960924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958313    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:21.960924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958317    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:21.960924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958322    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:21.960924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958326    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:21.960924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958330    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:21.960924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958334    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:21.960924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958338    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:21.960924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958359    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:21.960924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958373    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:21.960924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958377    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:21.960924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958382    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958386    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958391    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958396    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958400    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958404    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958408    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958412    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958416    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958420    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958425    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958429    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958434    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958438    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958443    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958448    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958452    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.958456    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959780    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959794    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:21.961769 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959799    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959804    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959809    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959815    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959819    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959824    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959828    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959833    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959838    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959842    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959847    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959853    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959858    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959862    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959868    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959872    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959876    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959881    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959885    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959889    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:21.962630 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959893    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959898    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959902    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959907    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959912    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959916    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959922    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959928    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959933    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959937    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959942    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959946    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959950    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959954    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959959    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959963    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959967    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959971    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959975    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959979    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:21.963422 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959984    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959989    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959993    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.959999    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960005    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960009    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960014    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960018    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960022    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960026    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960031    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960035    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960039    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960043    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960048    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960052    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960056    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960060    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960065    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960069    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:21.963950 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960073    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960078    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960082    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960086    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960090    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960095    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960100    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960104    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960108    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960112    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960116    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960120    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960124    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960128    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960132    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960137    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960141    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960146    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960153    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960157    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:21.964538 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960161    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960166    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960170    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960174    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960178    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960182    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960186    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960191    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960195    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960200    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960207    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960213    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960219    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960223    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960228    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960233    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960237    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960939    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960947    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960952    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:21.965053 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960956    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:21.965600 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960960    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:21.965600 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960964    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:21.965600 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960968    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:21.965600 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960980    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:21.965600 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960986    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:21.965600 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960992    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:21.965600 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.960997    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:21.965600 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961001    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:21.965600 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961007    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:21.965600 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961012    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:21.965600 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961016    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:21.965600 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961020    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:21.965600 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961025    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:21.965600 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961030    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:21.965600 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961034    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:21.965600 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961038    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:21.965600 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961042    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:21.965600 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961047    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:21.965600 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961051    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:21.966254 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961056    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:21.966254 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961061    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:21.966254 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961065    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:21.966254 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961069    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:21.966254 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961074    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:21.966254 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961078    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:21.966254 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961084    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:21.966254 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961088    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:21.966254 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961092    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:21.966254 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961097    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:21.966254 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961101    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:21.966254 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961106    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:21.966254 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961110    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:21.966254 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961114    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:21.966254 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961118    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:21.966254 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961122    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:21.966254 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961126    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:21.966254 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961132    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:21.966254 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961138    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961143    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961148    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961152    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961157    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961161    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961165    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961169    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961173    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961180    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961185    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961190    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961195    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961200    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961205    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961210    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961215    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961219    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961223    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961228    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:21.966751 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961232    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961236    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961240    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961244    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961248    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961252    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961256    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961260    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961266    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961270    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961274    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961278    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961282    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961286    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961291    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961295    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961299    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961304    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961308    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961312    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:21.967258 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961316    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961320    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961324    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961329    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961333    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961338    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961361    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961366    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961371    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961375    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961379    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961384    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961388    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961392    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961397    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961401    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961405    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.961409    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961523    2565 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961584    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:39:21.967877 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961596    2565 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961603    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961610    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961615    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961622    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961630    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961635    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961640    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961646    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961652    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961658    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961663    2565 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961668    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961673    2565 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961677    2565 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961683    2565 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961690    2565 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961695    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961700    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961708    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961714    2565 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961720    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961725    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:39:21.968400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961730    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961736    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961741    2565 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961746    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961751    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961756    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961761    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961768    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961772    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961777    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961782    2565 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961786    2565 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961791    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961799    2565 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961804    2565 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961809    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961815    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961819    2565 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961825    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961830    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961836    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961841    2565 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961846    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961851    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961856    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:39:21.968979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961861    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961866    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961871    2565 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961877    2565 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961884    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961889    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961894    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961899    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961904    2565 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961909    2565 flags.go:64] FLAG: --help="false"
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961914    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-130-204.ec2.internal"
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961919    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961924    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961929    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961935    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961940    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961945    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961949    2565 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961954    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961959    2565 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961964    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961969    2565 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961974    2565 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961981    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:39:21.969658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961986    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961991    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.961995    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962000    2565 flags.go:64] FLAG: --lock-file=""
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962006    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962012    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962017    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962025    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962030    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962035    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962039    2565 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962046    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962051    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962056    2565 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962061    2565 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962069    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962074    2565 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962081    2565 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962085    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962091    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962095    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962100    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962105    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962110    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962115    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:39:21.970259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962127    2565 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962132    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962138    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962143    2565 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962148    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962157    2565 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962161    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962168    2565 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962173    2565 flags.go:64] FLAG: --port="10250"
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962179    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962183    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0322607246fb40d6f"
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962189    2565 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962194    2565 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962199    2565 flags.go:64] FLAG: --register-node="true"
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962204    2565 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962210    2565 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962215    2565 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962220    2565 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962230    2565 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962236    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962240    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962245    2565 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962250    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962255    2565 flags.go:64] FLAG: --runonce="false"
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962260    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:39:21.970986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962265    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962270    2565 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962275    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962279    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962284    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962289    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962294    2565 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962299    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962304    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962309    2565 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962313    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962318    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962323    2565 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962328    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962339    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962362    2565 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962368    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962374    2565 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962379    2565 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962383    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962395    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962401    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962406    2565 flags.go:64] FLAG: --v="2"
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962413    2565 flags.go:64] FLAG: --version="false"
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962420    2565 flags.go:64] FLAG: --vmodule=""
Sep 16 18:39:21.971629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962428    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:39:21.972270 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.962433    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:39:21.972270 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962645    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:21.972270 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962652    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:21.972270 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962657    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:21.972270 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962664    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:21.972270 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962671    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:21.972270 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962675    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:21.972270 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962680    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:21.972270 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962684    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:21.972270 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962688    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:21.972270 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962692    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:21.972270 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962696    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:21.972270 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962700    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:21.972270 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962705    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:21.972270 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962709    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:21.972270 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962713    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:21.972270 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962717    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:21.972270 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962721    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:21.972270 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962728    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962733    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962740    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962744    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962750    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962755    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962760    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962765    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962769    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962782    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962787    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962792    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962797    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962803    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962807    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962812    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962817    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962821    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962825    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962830    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:21.972776 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962834    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962839    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962843    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962847    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962851    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962855    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962860    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962864    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962868    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962872    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962877    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962881    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962885    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962889    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962895    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962899    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962904    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962910    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962917    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962922    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:21.973306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962927    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962931    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962947    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962952    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962956    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962963    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962967    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962972    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962976    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962980    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962984    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962988    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962993    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.962997    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963001    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963006    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963010    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963015    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963019    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963023    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:21.973813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963027    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:21.974313 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963031    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:21.974313 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963035    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:21.974313 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963040    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:21.974313 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963044    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:21.974313 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963048    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:21.974313 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963054    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:21.974313 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963058    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:21.974313 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963062    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:21.974313 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963066    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:21.974313 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963070    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:21.974313 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963074    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:21.974313 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963078    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:21.974313 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963083    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:21.974313 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963087    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:21.974313 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963099    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:21.974313 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963104    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:21.974313 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963110    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:21.974313 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963114    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:21.974313 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963118    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:21.974797 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963122    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:21.974797 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.963126    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:21.974797 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.963914    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:39:21.974797 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.972541    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:39:21.974797 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.972560    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:39:21.974797 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.972581    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:39:21.974797 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.972587    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:39:21.974797 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972649    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:21.974797 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972654    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:21.974797 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972658    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:21.974797 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972661    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:21.974797 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972664    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:21.974797 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972667    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:21.974797 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972670    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:21.974797 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972673    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:21.975179 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972676    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:21.975179 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972679    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:21.975179 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972682    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:21.975179 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972686    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:21.975179 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972690    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:21.975179 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972693    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:21.975179 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972696    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:21.975179 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972699    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:21.975179 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972702    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:21.975179 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972705    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:21.975179 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972707    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:21.975179 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972711    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:21.975179 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972717    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:21.975179 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972720    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:21.975179 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972723    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:21.975179 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972726    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:21.975179 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972729    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:21.975179 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972731    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:21.975179 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972734    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:21.975661 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972737    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:21.975661 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972739    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:21.975661 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972743    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:21.975661 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972746    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:21.975661 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972748    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:21.975661 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972751    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:21.975661 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972754    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:21.975661 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972756    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:21.975661 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972759    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:21.975661 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972762    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:21.975661 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972764    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:21.975661 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972767    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:21.975661 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972770    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:21.975661 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972773    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:21.975661 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972777    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:21.975661 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972781    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:21.975661 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972784    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:21.975661 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972787    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:21.975661 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972790    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972792    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972795    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972798    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972800    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972803    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972805    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972808    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972811    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972814    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972817    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972819    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972822    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972824    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972827    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972830    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972833    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972840    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972842    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972845    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972848    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:21.976121 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972850    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972853    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972858    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972861    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972865    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972868    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972871    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972874    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972877    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972879    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972882    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972885    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972887    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972890    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972892    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972895    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972898    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972900    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972903    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972907    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:21.976706 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972915    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:21.977200 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972918    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:21.977200 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972921    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:21.977200 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972924    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:21.977200 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972926    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:21.977200 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972929    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:21.977200 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972931    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:21.977200 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972934    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:21.977200 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972937    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:21.977200 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972940    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:21.977200 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972943    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:21.977200 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972946    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:21.977200 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.972948    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:21.977200 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.972954    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:39:21.977200 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973092    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973098    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973101    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973104    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973107    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973109    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973112    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973115    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973117    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973120    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973123    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973125    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973128    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973131    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973133    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973136    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973139    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973144    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973148    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973152    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:21.977566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973154    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:21.978051 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973157    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:21.978051 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973161    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:21.978051 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973164    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:21.978051 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973167    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:21.978051 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973170    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:21.978051 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973173    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:21.978051 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973175    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:21.978051 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973179    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:21.978051 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973181    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:21.978051 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973184    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:21.978051 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973187    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:21.978051 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973191    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:21.978051 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973194    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:21.978051 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973197    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:21.978051 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973200    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:21.978051 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973204    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:21.978051 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973207    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:21.978051 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973210    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:21.978051 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973213    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:21.978576 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973216    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:21.978576 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973219    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:21.978576 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973221    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:21.978576 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973224    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:21.978576 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973227    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:21.978576 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973229    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:21.978576 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973232    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:21.978576 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973235    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:21.978576 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973237    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:21.978576 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973240    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:21.978576 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973243    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:21.978576 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973246    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:21.978576 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973249    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:21.978576 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973252    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:21.978576 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973256    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:21.978576 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973258    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:21.978576 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973261    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:21.978576 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973264    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:21.978576 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973266    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:21.979032 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973269    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:21.979032 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973272    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:21.979032 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973274    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:21.979032 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973277    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:21.979032 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973280    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:21.979032 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973284    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:21.979032 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973286    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:21.979032 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973289    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:21.979032 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973292    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:21.979032 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973294    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:21.979032 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973297    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:21.979032 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973300    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:21.979032 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973302    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:21.979032 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973305    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:21.979032 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973308    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:21.979032 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973310    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:21.979032 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973313    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:21.979032 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973316    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:21.979032 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973319    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973321    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973324    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973327    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973329    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973332    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973335    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973337    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973340    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973360    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973363    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973365    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973368    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973371    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973374    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973377    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973379    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973382    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973385    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973388    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:21.979540 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973390    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:21.980020 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:21.973393    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:21.980020 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.973398    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:39:21.980020 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.973529    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:39:21.980020 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.976944    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:39:21.980020 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.978078    2565 server.go:1019] "Starting client certificate rotation"
Sep 16 18:39:21.980020 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.978186    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:39:21.980020 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.978230    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:39:21.980945 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.980930    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:39:21.982175 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.982155    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:39:21.995645 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:21.995618    2565 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:39:22.000367 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.000333    2565 log.go:25] "Validated CRI v1 image API"
Sep 16 18:39:22.002218 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.002198    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:39:22.006597 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.006577    2565 fs.go:135] Filesystem UUIDs: map[53cbe09e-935f-4d0e-ac96-82929be8bb91:/dev/nvme0n1p3 6db667ec-816e-468b-ac8c-bfa13497e859:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 16 18:39:22.006665 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.006597    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:39:22.007581 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.007563    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:39:22.013014 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.012888    2565 manager.go:216] Machine: {Timestamp:2026-09-16 18:39:22.01141887 +0000 UTC m=+0.370936856 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099828 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a2e5a2e77aabfcd416b961dff1965 SystemUUID:ec2a2e5a-2e77-aabf-cd41-6b961dff1965 BootID:0fa948ad-a5ca-47c0-b3f6-a893bec4a6e0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e6:1a:01:e5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e6:1a:01:e5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:7e:19:be:7d:c7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:39:22.013014 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.013002    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:39:22.013179 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.013138    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:39:22.016622 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.016588    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:39:22.016776 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.016624    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-204.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:39:22.017511 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.017499    2565 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:39:22.017548 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.017512    2565 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:39:22.017548 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.017521    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:39:22.017548 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.017540    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:39:22.018827 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.018816    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:39:22.024377 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.024361    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:39:22.024516 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.024506    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:39:22.024560 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.024537    2565 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:39:22.024560 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.024553    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:39:22.024626 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.024568    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:39:22.024626 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.024580    2565 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:39:22.024626 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.024590    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:39:22.024999 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.024987    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:39:22.025035 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.025006    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:39:22.027487 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.027465    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:39:22.030505 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.030490    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:39:22.030585 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.030520    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:39:22.032636 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.032621    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:39:22.032704 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.032643    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:39:22.032704 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.032651    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:39:22.032704 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.032659    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:39:22.032704 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.032670    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:39:22.032704 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.032682    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:39:22.032704 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.032701    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:39:22.032704 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.032707    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:39:22.032960 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.032715    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:39:22.032960 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.032720    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:39:22.032960 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.032731    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:39:22.032960 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.032736    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:39:22.033690 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.033674    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:39:22.033690 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.033688    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:39:22.034905 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.034874    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:39:22.034905 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.034892    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-204.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:39:22.035779 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.035764    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ddnbr"
Sep 16 18:39:22.036976 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.036956    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-204.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:39:22.037853 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.037840    2565 server.go:1268] "Started kubelet"
Sep 16 18:39:22.037980 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.037945    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:39:22.038749 ip-10-0-130-204 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:39:22.039834 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.038832    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:39:22.039834 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.039386    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:39:22.044197 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.044170    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:39:22.045038 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.045019    2565 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:39:22.045366 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.045327    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:39:22.045606 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.045582    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ddnbr"
Sep 16 18:39:22.049074 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.047927    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-204.ec2.internal.18d5e121c0ffeb1a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-204.ec2.internal,UID:ip-10-0-130-204.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-204.ec2.internal,},FirstTimestamp:2026-09-16 18:39:22.037807898 +0000 UTC m=+0.397325980,LastTimestamp:2026-09-16 18:39:22.037807898 +0000 UTC m=+0.397325980,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-204.ec2.internal,}"
Sep 16 18:39:22.053758 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.053642    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:39:22.054868 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.054324    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:39:22.054868 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.054562    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-204.ec2.internal\" not found"
Sep 16 18:39:22.055203 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.055156    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:39:22.055297 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.055281    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:39:22.055468 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.055449    2565 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:39:22.055468 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.055465    2565 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:39:22.055581 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.055158    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:39:22.055650 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.055630    2565 factory.go:55] Registering systemd factory
Sep 16 18:39:22.055753 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.055676    2565 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:39:22.055932 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.055910    2565 factory.go:153] Registering CRI-O factory
Sep 16 18:39:22.055932 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.055926    2565 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:39:22.056076 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.056065    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:39:22.056139 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.056097    2565 factory.go:103] Registering Raw factory
Sep 16 18:39:22.056139 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.056117    2565 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:39:22.056452 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.056414    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-204.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:39:22.056608 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.056595    2565 manager.go:349] Starting recovery of all containers
Sep 16 18:39:22.068421 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.068401    2565 manager.go:354] Recovery completed
Sep 16 18:39:22.068747 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.068725    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:22.072904 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.072888    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:22.074704 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.074683    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:22.074785 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.074720    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:22.074785 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.074736    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:22.075319 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.075302    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:39:22.075319 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.075316    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:39:22.075472 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.075335    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:39:22.075472 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.075370    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:22.076585 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.076564    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:22.076672 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.076599    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:22.076672 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.076613    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:22.077611 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.077593    2565 policy_none.go:49] "None policy: Start"
Sep 16 18:39:22.077611 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.077611    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:39:22.077751 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.077621    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:39:22.079611 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.079596    2565 policy_none.go:47] "Start"
Sep 16 18:39:22.155528 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.155493    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-204.ec2.internal\" not found"
Sep 16 18:39:22.199055 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.199006    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:39:22.200302 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.200286    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:39:22.200409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.200373    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:39:22.200409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.200406    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:39:22.200511 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.200457    2565 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 16 18:39:22.203131 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.203046    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:22.247008 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.246986    2565 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:39:22.247132 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.247082    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:39:22.247132 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.247093    2565 server.go:87] "Starting device plugin registration server"
Sep 16 18:39:22.247448 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.247432    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:39:22.247521 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.247452    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:39:22.247571 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.247539    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:39:22.247630 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.247619    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:39:22.247677 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.247630    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:39:22.248420 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.248394    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:39:22.248513 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.248457    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-204.ec2.internal\" not found"
Sep 16 18:39:22.263219 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.263181    2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-204.ec2.internal\" not found" node="ip-10-0-130-204.ec2.internal"
Sep 16 18:39:22.301314 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.301265    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-204.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal"]
Sep 16 18:39:22.301411 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.301391    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:22.302520 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.302501    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:22.302640 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.302533    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:22.302640 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.302544    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:22.305020 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.305007    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:22.305244 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.305198    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-204.ec2.internal"
Sep 16 18:39:22.305244 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.305228    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:22.305866 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.305846    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:22.305955 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.305878    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:22.305955 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.305888    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:22.305955 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.305846    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:22.306087 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.305961    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:22.306087 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.305979    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:22.308258 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.308240    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal"
Sep 16 18:39:22.308401 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.308273    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:22.309113 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.309095    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:22.309212 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.309125    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:22.309212 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.309140    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:22.342427 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.342393    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-204.ec2.internal\" not found" node="ip-10-0-130-204.ec2.internal"
Sep 16 18:39:22.347314 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.347294    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-204.ec2.internal\" not found" node="ip-10-0-130-204.ec2.internal"
Sep 16 18:39:22.348261 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.348246    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:22.349201 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.349186    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:22.349308 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.349218    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:22.349308 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.349233    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:22.349308 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.349264    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-204.ec2.internal"
Sep 16 18:39:22.355549 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.355533    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-204.ec2.internal"
Sep 16 18:39:22.355615 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.355555    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-204.ec2.internal\": node \"ip-10-0-130-204.ec2.internal\" not found"
Sep 16 18:39:22.357015 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.356998    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/832091a938dd3fb43132057170181abf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal\" (UID: \"832091a938dd3fb43132057170181abf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal"
Sep 16 18:39:22.357073 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.357023    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1152677edddc8ea58737be1eaabf066e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-204.ec2.internal\" (UID: \"1152677edddc8ea58737be1eaabf066e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-204.ec2.internal"
Sep 16 18:39:22.357073 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.357051    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/832091a938dd3fb43132057170181abf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal\" (UID: \"832091a938dd3fb43132057170181abf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal"
Sep 16 18:39:22.373752 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.373717    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-204.ec2.internal\" not found"
Sep 16 18:39:22.457323 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.457240    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1152677edddc8ea58737be1eaabf066e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-204.ec2.internal\" (UID: \"1152677edddc8ea58737be1eaabf066e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-204.ec2.internal"
Sep 16 18:39:22.457323 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.457281    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/832091a938dd3fb43132057170181abf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal\" (UID: \"832091a938dd3fb43132057170181abf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal"
Sep 16 18:39:22.457323 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.457302    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/832091a938dd3fb43132057170181abf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal\" (UID: \"832091a938dd3fb43132057170181abf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal"
Sep 16 18:39:22.457546 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.457399    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/832091a938dd3fb43132057170181abf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal\" (UID: \"832091a938dd3fb43132057170181abf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal"
Sep 16 18:39:22.457546 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.457407    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/832091a938dd3fb43132057170181abf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal\" (UID: \"832091a938dd3fb43132057170181abf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal"
Sep 16 18:39:22.457546 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.457407    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1152677edddc8ea58737be1eaabf066e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-204.ec2.internal\" (UID: \"1152677edddc8ea58737be1eaabf066e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-204.ec2.internal"
Sep 16 18:39:22.474675 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.474633    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-204.ec2.internal\" not found"
Sep 16 18:39:22.575972 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.575915    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-204.ec2.internal\" not found"
Sep 16 18:39:22.645402 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.645340    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-204.ec2.internal"
Sep 16 18:39:22.649969 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.649944    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal"
Sep 16 18:39:22.676979 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.676944    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-204.ec2.internal\" not found"
Sep 16 18:39:22.778589 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.778497    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-204.ec2.internal\" not found"
Sep 16 18:39:22.879838 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.879806    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-204.ec2.internal\" not found"
Sep 16 18:39:22.977780 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.977742    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:39:22.978316 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.977947    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:39:22.978316 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:22.977947    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:39:22.981052 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:22.981026    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-204.ec2.internal\" not found"
Sep 16 18:39:23.011051 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:23.011016    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:23.053033 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:23.052930    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:34:22 +0000 UTC" deadline="2028-06-13 03:25:58.877574364 +0000 UTC"
Sep 16 18:39:23.053033 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:23.052971    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15248h46m35.824606795s"
Sep 16 18:39:23.082413 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:23.082365    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-204.ec2.internal\" not found"
Sep 16 18:39:23.183247 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:23.183063    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-204.ec2.internal\" not found"
Sep 16 18:39:23.284155 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:23.284113    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-204.ec2.internal\" not found"
Sep 16 18:39:23.292681 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:23.292650    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:23.346823 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:23.346776    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1152677edddc8ea58737be1eaabf066e.slice/crio-dcbdafb53fe92904f31e3ee8325f2012a34fb090a6d968faa429d00f08d91bee WatchSource:0}: Error finding container dcbdafb53fe92904f31e3ee8325f2012a34fb090a6d968faa429d00f08d91bee: Status 404 returned error can't find the container with id dcbdafb53fe92904f31e3ee8325f2012a34fb090a6d968faa429d00f08d91bee
Sep 16 18:39:23.352814 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:23.351913    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:39:23.354584 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:23.354566    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal"
Sep 16 18:39:23.366967 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:23.366931    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:39:23.367870 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:23.367852    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-204.ec2.internal"
Sep 16 18:39:23.375666 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:23.375641    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:39:23.409010 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:23.408971    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832091a938dd3fb43132057170181abf.slice/crio-2784d37270459609474258a0fb52e66e7eaa9daa7d860e807f334caa0ab77c34 WatchSource:0}: Error finding container 2784d37270459609474258a0fb52e66e7eaa9daa7d860e807f334caa0ab77c34: Status 404 returned error can't find the container with id 2784d37270459609474258a0fb52e66e7eaa9daa7d860e807f334caa0ab77c34
Sep 16 18:39:23.966184 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:23.966148    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:24.025926 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.025886    2565 apiserver.go:52] "Watching apiserver"
Sep 16 18:39:24.033271 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.033236    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:39:24.033834 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.033808    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal","openshift-multus/network-metrics-daemon-7r6xr","openshift-cluster-node-tuning-operator/tuned-zj4zw","openshift-dns/node-resolver-tch8d","openshift-multus/multus-8dncf","openshift-multus/multus-additional-cni-plugins-hhl4g","openshift-network-diagnostics/network-check-target-hqh7p","openshift-network-operator/iptables-alerter-r596t","openshift-ovn-kubernetes/ovnkube-node-lpp26","kube-system/global-pull-secret-syncer-ddtnd","kube-system/konnectivity-agent-t9j4z","kube-system/kube-apiserver-proxy-ip-10-0-130-204.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb","openshift-image-registry/node-ca-8pj92"]
Sep 16 18:39:24.036602 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.036578    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:24.036730 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:24.036668    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqh7p" podUID="dbb3e980-5ba7-4b12-82b2-1fa036ec00b7"
Sep 16 18:39:24.038808 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.038784    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.041685 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.041658    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.043262 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.043238    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tch8d"
Sep 16 18:39:24.043528 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.043506    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:39:24.044016 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.043853    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:39:24.044016 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.043931    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:39:24.044736 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.044490    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f2jbf\""
Sep 16 18:39:24.047677 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.046449    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:39:24.047677 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.046554    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.047677 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.047025    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:39:24.047677 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.047290    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:39:24.047677 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.047537    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jtwtb\""
Sep 16 18:39:24.049642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.048470    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:39:24.049642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.048492    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5nvj7\""
Sep 16 18:39:24.049642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.049523    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.051123 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.051101    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:39:24.052013 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.051253    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:39:24.052013 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.051756    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.052745 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.052724    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:39:24.053008 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.052990    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:39:24.053196 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.053176    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-p4cc5\""
Sep 16 18:39:24.053984 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.053923    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:39:24.054486 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.054465    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8pj92"
Sep 16 18:39:24.056833 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.056809    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:24.056995 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:24.056903    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7r6xr" podUID="2344014b-38cc-448f-9c9d-6b49d8e80b85"
Sep 16 18:39:24.057283 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.057263    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:39:24.057283 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.057276    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:39:24.058522 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.058502    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:39:24.058784 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.058740    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6jwkz\""
Sep 16 18:39:24.059227 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.059211    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:39:24.059440 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.059417    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:39:24.059524 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.059474    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:39:24.059577 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.059421    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:39:24.059777 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.059756    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:24.059868 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:24.059834    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddtnd" podUID="cea41c21-51ad-46c1-8f7c-fc4dab1008e7"
Sep 16 18:39:24.060215 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.060195    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:39:24.060710 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.060687    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6rjpw\""
Sep 16 18:39:24.061599 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.060983    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-whsh9\""
Sep 16 18:39:24.061599 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.061176    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:39:24.061599 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.061325    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:39:24.061599 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.061459    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:39:24.063118 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.062101    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t9j4z"
Sep 16 18:39:24.064889 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.064429    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r596t"
Sep 16 18:39:24.065307 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.064990    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-f94h2\""
Sep 16 18:39:24.065307 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.065204    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:39:24.065307 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.065255    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:39:24.066632 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.066616    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:39:24.067433 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067410    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqfsl\" (UniqueName: \"kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl\") pod \"network-check-target-hqh7p\" (UID: \"dbb3e980-5ba7-4b12-82b2-1fa036ec00b7\") " pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:24.067534 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067447    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-system-cni-dir\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.067534 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067471    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-multus-conf-dir\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.067534 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-systemd\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.067699 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067524    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-cnibin\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.067699 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067563    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-host-run-k8s-cni-cncf-io\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.067699 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067588    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f251ea42-1492-4bf4-942e-fdd522e5bbc5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.067699 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067612    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-lib-modules\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.067699 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067667    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-multus-socket-dir-parent\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.067985 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067698    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2dce3890-5613-4464-91b5-388c8579cfee-multus-daemon-config\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.067985 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067726    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c940a913-b6ea-4d59-9e84-60a83842c3a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.067985 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067751    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxp77\" (UniqueName: \"kubernetes.io/projected/2344014b-38cc-448f-9c9d-6b49d8e80b85-kube-api-access-sxp77\") pod \"network-metrics-daemon-7r6xr\" (UID: \"2344014b-38cc-448f-9c9d-6b49d8e80b85\") " pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:24.067985 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067785    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-hostroot\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.067985 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067809    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-systemd-units\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.067985 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067829    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-etc-openvswitch\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.067985 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067855    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-run-ovn\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.067985 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067878    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-var-lib-kubelet\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.067985 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067900    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f251ea42-1492-4bf4-942e-fdd522e5bbc5-cnibin\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.067985 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067927    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-kubelet\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.068480 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.067989    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1cdba36-53e5-400a-9fdd-7c62a3fe0086-host\") pod \"node-ca-8pj92\" (UID: \"d1cdba36-53e5-400a-9fdd-7c62a3fe0086\") " pod="openshift-image-registry/node-ca-8pj92"
Sep 16 18:39:24.068480 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068019    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-sysconfig\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.068480 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068054    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-run-netns\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.068480 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068087    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-run-systemd\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.068480 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068113    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bll2v\" (UniqueName: \"kubernetes.io/projected/d1cdba36-53e5-400a-9fdd-7c62a3fe0086-kube-api-access-bll2v\") pod \"node-ca-8pj92\" (UID: \"d1cdba36-53e5-400a-9fdd-7c62a3fe0086\") " pod="openshift-image-registry/node-ca-8pj92"
Sep 16 18:39:24.068480 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068139    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe676eb4-48c2-485b-9752-2198abad740a-tmp-dir\") pod \"node-resolver-tch8d\" (UID: \"fe676eb4-48c2-485b-9752-2198abad740a\") " pod="openshift-dns/node-resolver-tch8d"
Sep 16 18:39:24.068480 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068163    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-cni-netd\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.068480 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068202    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.068480 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068266    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-tuned\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.068480 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068291    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-multus-cni-dir\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.068480 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068314    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-host\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.068480 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068363    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f251ea42-1492-4bf4-942e-fdd522e5bbc5-system-cni-dir\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.068480 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068387    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c940a913-b6ea-4d59-9e84-60a83842c3a1-socket-dir\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.069068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068495    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-kubernetes\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.069068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068562    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-host-var-lib-kubelet\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.069068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068584    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr7kj\" (UniqueName: \"kubernetes.io/projected/2dce3890-5613-4464-91b5-388c8579cfee-kube-api-access-zr7kj\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.069068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068605    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c940a913-b6ea-4d59-9e84-60a83842c3a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.069068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068626    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d1cdba36-53e5-400a-9fdd-7c62a3fe0086-serviceca\") pod \"node-ca-8pj92\" (UID: \"d1cdba36-53e5-400a-9fdd-7c62a3fe0086\") " pod="openshift-image-registry/node-ca-8pj92"
Sep 16 18:39:24.069068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068663    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-sysctl-conf\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.069068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068685    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-host-var-lib-cni-multus\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.069068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068727    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-cni-bin\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.069068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068751    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7872de0b-da8d-4589-afaa-8675716cd80b-ovnkube-script-lib\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.069068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068782    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs\") pod \"network-metrics-daemon-7r6xr\" (UID: \"2344014b-38cc-448f-9c9d-6b49d8e80b85\") " pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:24.069068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068803    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-sysctl-d\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.069068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068823    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5368e2c9-26dd-429b-827a-270b3c2fe324-tmp\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.069068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068854    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n74hb\" (UniqueName: \"kubernetes.io/projected/fe676eb4-48c2-485b-9752-2198abad740a-kube-api-access-n74hb\") pod \"node-resolver-tch8d\" (UID: \"fe676eb4-48c2-485b-9752-2198abad740a\") " pod="openshift-dns/node-resolver-tch8d"
Sep 16 18:39:24.069068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068855    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:39:24.069068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068882    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:39:24.069068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068899    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f251ea42-1492-4bf4-942e-fdd522e5bbc5-cni-binary-copy\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.069068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068931    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-slash\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.069068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068964    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-run-openvswitch\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.069920 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.068998    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-run-ovn-kubernetes\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.069920 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069026    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-host-var-lib-cni-bin\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.069920 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069053    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-node-log\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.069920 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069075    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c940a913-b6ea-4d59-9e84-60a83842c3a1-registration-dir\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.069920 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069099    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c940a913-b6ea-4d59-9e84-60a83842c3a1-device-dir\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.069920 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069131    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqqzj\" (UniqueName: \"kubernetes.io/projected/c940a913-b6ea-4d59-9e84-60a83842c3a1-kube-api-access-wqqzj\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.069920 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-modprobe-d\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.069920 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069174    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-sys\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.069920 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069213    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fe676eb4-48c2-485b-9752-2198abad740a-hosts-file\") pod \"node-resolver-tch8d\" (UID: \"fe676eb4-48c2-485b-9752-2198abad740a\") " pod="openshift-dns/node-resolver-tch8d"
Sep 16 18:39:24.069920 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069228    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-etc-kubernetes\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.069920 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069265    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-prqfq\""
Sep 16 18:39:24.069920 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069255    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f251ea42-1492-4bf4-942e-fdd522e5bbc5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.069920 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069314    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f251ea42-1492-4bf4-942e-fdd522e5bbc5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.069920 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069367    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-var-lib-openvswitch\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.069920 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069396    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-log-socket\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.069920 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069439    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2dce3890-5613-4464-91b5-388c8579cfee-cni-binary-copy\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.069920 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069475    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-host-run-netns\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.070818 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069505    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-host-run-multus-certs\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.070818 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069528    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7872de0b-da8d-4589-afaa-8675716cd80b-ovnkube-config\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.070818 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069549    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7872de0b-da8d-4589-afaa-8675716cd80b-env-overrides\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.070818 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069566    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7872de0b-da8d-4589-afaa-8675716cd80b-ovn-node-metrics-cert\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.070818 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069580    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf6km\" (UniqueName: \"kubernetes.io/projected/7872de0b-da8d-4589-afaa-8675716cd80b-kube-api-access-mf6km\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.070818 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069594    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c940a913-b6ea-4d59-9e84-60a83842c3a1-sys-fs\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.070818 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069609    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-run\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.070818 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069637    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t876q\" (UniqueName: \"kubernetes.io/projected/5368e2c9-26dd-429b-827a-270b3c2fe324-kube-api-access-t876q\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.070818 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.069670    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78wql\" (UniqueName: \"kubernetes.io/projected/f251ea42-1492-4bf4-942e-fdd522e5bbc5-kube-api-access-78wql\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.072402 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.072382    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:39:24.100083 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.100050    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cd2k2"
Sep 16 18:39:24.110117 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.110089    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cd2k2"
Sep 16 18:39:24.127236 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.127203    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:24.156299 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.156262    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:39:24.170246 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqfsl\" (UniqueName: \"kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl\") pod \"network-check-target-hqh7p\" (UID: \"dbb3e980-5ba7-4b12-82b2-1fa036ec00b7\") " pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:24.170246 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170254    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-system-cni-dir\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.170558 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170284    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-multus-conf-dir\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.170558 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170309    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-systemd\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.170558 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170337    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-cnibin\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.170558 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170380    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-host-run-k8s-cni-cncf-io\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.170558 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170409    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f251ea42-1492-4bf4-942e-fdd522e5bbc5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.170558 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170436    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-lib-modules\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.170558 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-multus-socket-dir-parent\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.170558 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170486    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2dce3890-5613-4464-91b5-388c8579cfee-multus-daemon-config\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.170558 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170512    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c940a913-b6ea-4d59-9e84-60a83842c3a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.170558 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170551    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-kubelet-config\") pod \"global-pull-secret-syncer-ddtnd\" (UID: \"cea41c21-51ad-46c1-8f7c-fc4dab1008e7\") " pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:24.170995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170581    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxp77\" (UniqueName: \"kubernetes.io/projected/2344014b-38cc-448f-9c9d-6b49d8e80b85-kube-api-access-sxp77\") pod \"network-metrics-daemon-7r6xr\" (UID: \"2344014b-38cc-448f-9c9d-6b49d8e80b85\") " pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:24.170995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-hostroot\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.170995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170665    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-hostroot\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.170995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170667    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-host-run-k8s-cni-cncf-io\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.170995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170784    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-multus-socket-dir-parent\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.170995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170809    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-multus-conf-dir\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.170995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170872    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-systemd\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.170995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170905    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-lib-modules\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.170995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170926    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-cnibin\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.170995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170927    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-system-cni-dir\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.170995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170975    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-systemd-units\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.171583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171004    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c940a913-b6ea-4d59-9e84-60a83842c3a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.171583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.170945    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-systemd-units\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.171583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171044    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-etc-openvswitch\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.171583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171072    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-run-ovn\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.171583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171102    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret\") pod \"global-pull-secret-syncer-ddtnd\" (UID: \"cea41c21-51ad-46c1-8f7c-fc4dab1008e7\") " pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:24.171583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171130    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ef3366e-85c0-4283-a4f0-aceaf04445a4-host-slash\") pod \"iptables-alerter-r596t\" (UID: \"9ef3366e-85c0-4283-a4f0-aceaf04445a4\") " pod="openshift-network-operator/iptables-alerter-r596t"
Sep 16 18:39:24.171583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171156    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-var-lib-kubelet\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.171583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171171    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-run-ovn\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.171583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171224    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f251ea42-1492-4bf4-942e-fdd522e5bbc5-cnibin\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.171583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171230    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-etc-openvswitch\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.171583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171240    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2dce3890-5613-4464-91b5-388c8579cfee-multus-daemon-config\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.171583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171181    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f251ea42-1492-4bf4-942e-fdd522e5bbc5-cnibin\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.171583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171279    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-var-lib-kubelet\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.171583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171294    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-kubelet\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.171583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1cdba36-53e5-400a-9fdd-7c62a3fe0086-host\") pod \"node-ca-8pj92\" (UID: \"d1cdba36-53e5-400a-9fdd-7c62a3fe0086\") " pod="openshift-image-registry/node-ca-8pj92"
Sep 16 18:39:24.171583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171376    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-kubelet\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.171583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-sysconfig\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.172409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171419    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-run-netns\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.172409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171425    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1cdba36-53e5-400a-9fdd-7c62a3fe0086-host\") pod \"node-ca-8pj92\" (UID: \"d1cdba36-53e5-400a-9fdd-7c62a3fe0086\") " pod="openshift-image-registry/node-ca-8pj92"
Sep 16 18:39:24.172409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171463    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-run-netns\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.172409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171490    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-sysconfig\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.172409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171455    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-run-systemd\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.172409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171521    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-run-systemd\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.172409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171533    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bll2v\" (UniqueName: \"kubernetes.io/projected/d1cdba36-53e5-400a-9fdd-7c62a3fe0086-kube-api-access-bll2v\") pod \"node-ca-8pj92\" (UID: \"d1cdba36-53e5-400a-9fdd-7c62a3fe0086\") " pod="openshift-image-registry/node-ca-8pj92"
Sep 16 18:39:24.172409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171533    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f251ea42-1492-4bf4-942e-fdd522e5bbc5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.172409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171582    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe676eb4-48c2-485b-9752-2198abad740a-tmp-dir\") pod \"node-resolver-tch8d\" (UID: \"fe676eb4-48c2-485b-9752-2198abad740a\") " pod="openshift-dns/node-resolver-tch8d"
Sep 16 18:39:24.172409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171610    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-cni-netd\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.172409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171638    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.172409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171664    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-tuned\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.172409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171675    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-cni-netd\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.172409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171687    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-multus-cni-dir\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.172409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171711    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-host\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.172409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171724    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.172409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171770    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f251ea42-1492-4bf4-942e-fdd522e5bbc5-system-cni-dir\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.173241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171797    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c940a913-b6ea-4d59-9e84-60a83842c3a1-socket-dir\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.173241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171825    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9ef3366e-85c0-4283-a4f0-aceaf04445a4-iptables-alerter-script\") pod \"iptables-alerter-r596t\" (UID: \"9ef3366e-85c0-4283-a4f0-aceaf04445a4\") " pod="openshift-network-operator/iptables-alerter-r596t"
Sep 16 18:39:24.173241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171850    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-kubernetes\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.173241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171855    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-host\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.173241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171855    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f251ea42-1492-4bf4-942e-fdd522e5bbc5-system-cni-dir\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.173241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171882    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-multus-cni-dir\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.173241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-host-var-lib-kubelet\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.173241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171909    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-kubernetes\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.173241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171939    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe676eb4-48c2-485b-9752-2198abad740a-tmp-dir\") pod \"node-resolver-tch8d\" (UID: \"fe676eb4-48c2-485b-9752-2198abad740a\") " pod="openshift-dns/node-resolver-tch8d"
Sep 16 18:39:24.173241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171949    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-host-var-lib-kubelet\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.173241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171978    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c940a913-b6ea-4d59-9e84-60a83842c3a1-socket-dir\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.173241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.171981    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr7kj\" (UniqueName: \"kubernetes.io/projected/2dce3890-5613-4464-91b5-388c8579cfee-kube-api-access-zr7kj\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.173241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172013    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c940a913-b6ea-4d59-9e84-60a83842c3a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.173241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172017    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:39:24.173241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172038    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d1cdba36-53e5-400a-9fdd-7c62a3fe0086-serviceca\") pod \"node-ca-8pj92\" (UID: \"d1cdba36-53e5-400a-9fdd-7c62a3fe0086\") " pod="openshift-image-registry/node-ca-8pj92"
Sep 16 18:39:24.173241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172083    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd830733-5618-401e-945e-21a97bb4b76f-agent-certs\") pod \"konnectivity-agent-t9j4z\" (UID: \"dd830733-5618-401e-945e-21a97bb4b76f\") " pod="kube-system/konnectivity-agent-t9j4z"
Sep 16 18:39:24.173241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172128    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-sysctl-conf\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.173241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172136    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c940a913-b6ea-4d59-9e84-60a83842c3a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.174141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172154    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-host-var-lib-cni-multus\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.174141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172180    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-cni-bin\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.174141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172203    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7872de0b-da8d-4589-afaa-8675716cd80b-ovnkube-script-lib\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.174141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172216    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-host-var-lib-cni-multus\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.174141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs\") pod \"network-metrics-daemon-7r6xr\" (UID: \"2344014b-38cc-448f-9c9d-6b49d8e80b85\") " pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:24.174141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172266    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-cni-bin\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.174141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172269    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-sysctl-conf\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.174141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172287    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-sysctl-d\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.174141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5368e2c9-26dd-429b-827a-270b3c2fe324-tmp\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.174141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n74hb\" (UniqueName: \"kubernetes.io/projected/fe676eb4-48c2-485b-9752-2198abad740a-kube-api-access-n74hb\") pod \"node-resolver-tch8d\" (UID: \"fe676eb4-48c2-485b-9752-2198abad740a\") " pod="openshift-dns/node-resolver-tch8d"
Sep 16 18:39:24.174141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172389    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f251ea42-1492-4bf4-942e-fdd522e5bbc5-cni-binary-copy\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.174141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172401    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-sysctl-d\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.174141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172420    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-slash\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.174141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172479    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-run-openvswitch\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.174141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172505    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-run-ovn-kubernetes\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.174141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172529    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-host-var-lib-cni-bin\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.174141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172553    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-node-log\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.174944 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172578    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c940a913-b6ea-4d59-9e84-60a83842c3a1-registration-dir\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.174944 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172588    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d1cdba36-53e5-400a-9fdd-7c62a3fe0086-serviceca\") pod \"node-ca-8pj92\" (UID: \"d1cdba36-53e5-400a-9fdd-7c62a3fe0086\") " pod="openshift-image-registry/node-ca-8pj92"
Sep 16 18:39:24.174944 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172605    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c940a913-b6ea-4d59-9e84-60a83842c3a1-device-dir\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.174944 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqqzj\" (UniqueName: \"kubernetes.io/projected/c940a913-b6ea-4d59-9e84-60a83842c3a1-kube-api-access-wqqzj\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.174944 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172658    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd830733-5618-401e-945e-21a97bb4b76f-konnectivity-ca\") pod \"konnectivity-agent-t9j4z\" (UID: \"dd830733-5618-401e-945e-21a97bb4b76f\") " pod="kube-system/konnectivity-agent-t9j4z"
Sep 16 18:39:24.174944 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172647    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-run-ovn-kubernetes\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.174944 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172683    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjkht\" (UniqueName: \"kubernetes.io/projected/9ef3366e-85c0-4283-a4f0-aceaf04445a4-kube-api-access-mjkht\") pod \"iptables-alerter-r596t\" (UID: \"9ef3366e-85c0-4283-a4f0-aceaf04445a4\") " pod="openshift-network-operator/iptables-alerter-r596t"
Sep 16 18:39:24.174944 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172714    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-modprobe-d\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.174944 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172736    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-host-var-lib-cni-bin\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.174944 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172739    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-sys\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.174944 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172779    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fe676eb4-48c2-485b-9752-2198abad740a-hosts-file\") pod \"node-resolver-tch8d\" (UID: \"fe676eb4-48c2-485b-9752-2198abad740a\") " pod="openshift-dns/node-resolver-tch8d"
Sep 16 18:39:24.174944 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172787    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-sys\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.174944 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172805    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-etc-kubernetes\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.174944 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172832    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f251ea42-1492-4bf4-942e-fdd522e5bbc5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.174944 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:24.172845    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:24.174944 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172858    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f251ea42-1492-4bf4-942e-fdd522e5bbc5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.174944 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172875    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7872de0b-da8d-4589-afaa-8675716cd80b-ovnkube-script-lib\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.174944 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172888    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-var-lib-openvswitch\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.175860 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:24.172938    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs podName:2344014b-38cc-448f-9c9d-6b49d8e80b85 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:24.672915675 +0000 UTC m=+3.032433663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs") pod "network-metrics-daemon-7r6xr" (UID: "2344014b-38cc-448f-9c9d-6b49d8e80b85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:24.175860 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172954    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-var-lib-openvswitch\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.175860 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-log-socket\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.175860 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.172985    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-node-log\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.175860 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173001    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2dce3890-5613-4464-91b5-388c8579cfee-cni-binary-copy\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.175860 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-host-run-netns\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.175860 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173047    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c940a913-b6ea-4d59-9e84-60a83842c3a1-registration-dir\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.175860 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173057    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-host-run-multus-certs\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.175860 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7872de0b-da8d-4589-afaa-8675716cd80b-ovnkube-config\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.175860 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173126    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7872de0b-da8d-4589-afaa-8675716cd80b-env-overrides\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.175860 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173153    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7872de0b-da8d-4589-afaa-8675716cd80b-ovn-node-metrics-cert\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.175860 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173178    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf6km\" (UniqueName: \"kubernetes.io/projected/7872de0b-da8d-4589-afaa-8675716cd80b-kube-api-access-mf6km\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.175860 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173182    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-run-openvswitch\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.175860 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173206    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c940a913-b6ea-4d59-9e84-60a83842c3a1-sys-fs\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.175860 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173225    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-host-slash\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.175860 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173233    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-run\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.175860 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173247    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f251ea42-1492-4bf4-942e-fdd522e5bbc5-cni-binary-copy\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.176695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173260    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t876q\" (UniqueName: \"kubernetes.io/projected/5368e2c9-26dd-429b-827a-270b3c2fe324-kube-api-access-t876q\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.176695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173287    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78wql\" (UniqueName: \"kubernetes.io/projected/f251ea42-1492-4bf4-942e-fdd522e5bbc5-kube-api-access-78wql\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.176695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173301    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-modprobe-d\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.176695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173301    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c940a913-b6ea-4d59-9e84-60a83842c3a1-device-dir\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.176695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173314    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-dbus\") pod \"global-pull-secret-syncer-ddtnd\" (UID: \"cea41c21-51ad-46c1-8f7c-fc4dab1008e7\") " pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:24.176695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173367    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fe676eb4-48c2-485b-9752-2198abad740a-hosts-file\") pod \"node-resolver-tch8d\" (UID: \"fe676eb4-48c2-485b-9752-2198abad740a\") " pod="openshift-dns/node-resolver-tch8d"
Sep 16 18:39:24.176695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173625    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c940a913-b6ea-4d59-9e84-60a83842c3a1-sys-fs\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.176695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173663    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5368e2c9-26dd-429b-827a-270b3c2fe324-run\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.176695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173775    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7872de0b-da8d-4589-afaa-8675716cd80b-log-socket\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.176695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173783    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7872de0b-da8d-4589-afaa-8675716cd80b-ovnkube-config\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.176695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173789    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7872de0b-da8d-4589-afaa-8675716cd80b-env-overrides\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.176695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173838    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-host-run-netns\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.176695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173864    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-etc-kubernetes\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.176695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173877    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2dce3890-5613-4464-91b5-388c8579cfee-host-run-multus-certs\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.176695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.173994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f251ea42-1492-4bf4-942e-fdd522e5bbc5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.176695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.174220    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f251ea42-1492-4bf4-942e-fdd522e5bbc5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.176695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.174295    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2dce3890-5613-4464-91b5-388c8579cfee-cni-binary-copy\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.176695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.176656    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5368e2c9-26dd-429b-827a-270b3c2fe324-tmp\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.177657 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.176724    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5368e2c9-26dd-429b-827a-270b3c2fe324-etc-tuned\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.177717 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:24.177678    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:24.177717 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:24.177698    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:24.177717 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:24.177713    2565 projected.go:196] Error preparing data for projected volume kube-api-access-xqfsl for pod openshift-network-diagnostics/network-check-target-hqh7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:24.177855 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:24.177774    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl podName:dbb3e980-5ba7-4b12-82b2-1fa036ec00b7 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:24.677756776 +0000 UTC m=+3.037274781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xqfsl" (UniqueName: "kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl") pod "network-check-target-hqh7p" (UID: "dbb3e980-5ba7-4b12-82b2-1fa036ec00b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:24.178571 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.178510    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7872de0b-da8d-4589-afaa-8675716cd80b-ovn-node-metrics-cert\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.180036 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.179977    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxp77\" (UniqueName: \"kubernetes.io/projected/2344014b-38cc-448f-9c9d-6b49d8e80b85-kube-api-access-sxp77\") pod \"network-metrics-daemon-7r6xr\" (UID: \"2344014b-38cc-448f-9c9d-6b49d8e80b85\") " pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:24.180400 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.180376    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bll2v\" (UniqueName: \"kubernetes.io/projected/d1cdba36-53e5-400a-9fdd-7c62a3fe0086-kube-api-access-bll2v\") pod \"node-ca-8pj92\" (UID: \"d1cdba36-53e5-400a-9fdd-7c62a3fe0086\") " pod="openshift-image-registry/node-ca-8pj92"
Sep 16 18:39:24.183506 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.183481    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n74hb\" (UniqueName: \"kubernetes.io/projected/fe676eb4-48c2-485b-9752-2198abad740a-kube-api-access-n74hb\") pod \"node-resolver-tch8d\" (UID: \"fe676eb4-48c2-485b-9752-2198abad740a\") " pod="openshift-dns/node-resolver-tch8d"
Sep 16 18:39:24.183850 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.183811    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr7kj\" (UniqueName: \"kubernetes.io/projected/2dce3890-5613-4464-91b5-388c8579cfee-kube-api-access-zr7kj\") pod \"multus-8dncf\" (UID: \"2dce3890-5613-4464-91b5-388c8579cfee\") " pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.186795 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.186764    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t876q\" (UniqueName: \"kubernetes.io/projected/5368e2c9-26dd-429b-827a-270b3c2fe324-kube-api-access-t876q\") pod \"tuned-zj4zw\" (UID: \"5368e2c9-26dd-429b-827a-270b3c2fe324\") " pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.187189 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.187163    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqqzj\" (UniqueName: \"kubernetes.io/projected/c940a913-b6ea-4d59-9e84-60a83842c3a1-kube-api-access-wqqzj\") pod \"aws-ebs-csi-driver-node-skqkb\" (UID: \"c940a913-b6ea-4d59-9e84-60a83842c3a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.187992 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.187969    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf6km\" (UniqueName: \"kubernetes.io/projected/7872de0b-da8d-4589-afaa-8675716cd80b-kube-api-access-mf6km\") pod \"ovnkube-node-lpp26\" (UID: \"7872de0b-da8d-4589-afaa-8675716cd80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.188336 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.188318    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78wql\" (UniqueName: \"kubernetes.io/projected/f251ea42-1492-4bf4-942e-fdd522e5bbc5-kube-api-access-78wql\") pod \"multus-additional-cni-plugins-hhl4g\" (UID: \"f251ea42-1492-4bf4-942e-fdd522e5bbc5\") " pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.205795 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.205725    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal" event={"ID":"832091a938dd3fb43132057170181abf","Type":"ContainerStarted","Data":"2784d37270459609474258a0fb52e66e7eaa9daa7d860e807f334caa0ab77c34"}
Sep 16 18:39:24.207288 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.207254    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-204.ec2.internal" event={"ID":"1152677edddc8ea58737be1eaabf066e","Type":"ContainerStarted","Data":"dcbdafb53fe92904f31e3ee8325f2012a34fb090a6d968faa429d00f08d91bee"}
Sep 16 18:39:24.273811 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.273729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret\") pod \"global-pull-secret-syncer-ddtnd\" (UID: \"cea41c21-51ad-46c1-8f7c-fc4dab1008e7\") " pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:24.273811 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.273766    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ef3366e-85c0-4283-a4f0-aceaf04445a4-host-slash\") pod \"iptables-alerter-r596t\" (UID: \"9ef3366e-85c0-4283-a4f0-aceaf04445a4\") " pod="openshift-network-operator/iptables-alerter-r596t"
Sep 16 18:39:24.274008 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.273838    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ef3366e-85c0-4283-a4f0-aceaf04445a4-host-slash\") pod \"iptables-alerter-r596t\" (UID: \"9ef3366e-85c0-4283-a4f0-aceaf04445a4\") " pod="openshift-network-operator/iptables-alerter-r596t"
Sep 16 18:39:24.274008 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:24.273890    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:24.274008 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.273902    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9ef3366e-85c0-4283-a4f0-aceaf04445a4-iptables-alerter-script\") pod \"iptables-alerter-r596t\" (UID: \"9ef3366e-85c0-4283-a4f0-aceaf04445a4\") " pod="openshift-network-operator/iptables-alerter-r596t"
Sep 16 18:39:24.274008 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.273940    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd830733-5618-401e-945e-21a97bb4b76f-agent-certs\") pod \"konnectivity-agent-t9j4z\" (UID: \"dd830733-5618-401e-945e-21a97bb4b76f\") " pod="kube-system/konnectivity-agent-t9j4z"
Sep 16 18:39:24.274008 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:24.273963    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret podName:cea41c21-51ad-46c1-8f7c-fc4dab1008e7 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:24.773940501 +0000 UTC m=+3.133458489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret") pod "global-pull-secret-syncer-ddtnd" (UID: "cea41c21-51ad-46c1-8f7c-fc4dab1008e7") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:24.274254 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.274014    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd830733-5618-401e-945e-21a97bb4b76f-konnectivity-ca\") pod \"konnectivity-agent-t9j4z\" (UID: \"dd830733-5618-401e-945e-21a97bb4b76f\") " pod="kube-system/konnectivity-agent-t9j4z"
Sep 16 18:39:24.274254 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.274125    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjkht\" (UniqueName: \"kubernetes.io/projected/9ef3366e-85c0-4283-a4f0-aceaf04445a4-kube-api-access-mjkht\") pod \"iptables-alerter-r596t\" (UID: \"9ef3366e-85c0-4283-a4f0-aceaf04445a4\") " pod="openshift-network-operator/iptables-alerter-r596t"
Sep 16 18:39:24.274254 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.274175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-dbus\") pod \"global-pull-secret-syncer-ddtnd\" (UID: \"cea41c21-51ad-46c1-8f7c-fc4dab1008e7\") " pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:24.274254 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.274242    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-kubelet-config\") pod \"global-pull-secret-syncer-ddtnd\" (UID: \"cea41c21-51ad-46c1-8f7c-fc4dab1008e7\") " pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:24.274483 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.274359    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-kubelet-config\") pod \"global-pull-secret-syncer-ddtnd\" (UID: \"cea41c21-51ad-46c1-8f7c-fc4dab1008e7\") " pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:24.274483 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.274401    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-dbus\") pod \"global-pull-secret-syncer-ddtnd\" (UID: \"cea41c21-51ad-46c1-8f7c-fc4dab1008e7\") " pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:24.274614 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.274590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9ef3366e-85c0-4283-a4f0-aceaf04445a4-iptables-alerter-script\") pod \"iptables-alerter-r596t\" (UID: \"9ef3366e-85c0-4283-a4f0-aceaf04445a4\") " pod="openshift-network-operator/iptables-alerter-r596t"
Sep 16 18:39:24.275233 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.275209    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd830733-5618-401e-945e-21a97bb4b76f-konnectivity-ca\") pod \"konnectivity-agent-t9j4z\" (UID: \"dd830733-5618-401e-945e-21a97bb4b76f\") " pod="kube-system/konnectivity-agent-t9j4z"
Sep 16 18:39:24.277430 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.277402    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd830733-5618-401e-945e-21a97bb4b76f-agent-certs\") pod \"konnectivity-agent-t9j4z\" (UID: \"dd830733-5618-401e-945e-21a97bb4b76f\") " pod="kube-system/konnectivity-agent-t9j4z"
Sep 16 18:39:24.286336 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.286303    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjkht\" (UniqueName: \"kubernetes.io/projected/9ef3366e-85c0-4283-a4f0-aceaf04445a4-kube-api-access-mjkht\") pod \"iptables-alerter-r596t\" (UID: \"9ef3366e-85c0-4283-a4f0-aceaf04445a4\") " pod="openshift-network-operator/iptables-alerter-r596t"
Sep 16 18:39:24.355109 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.355057    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb"
Sep 16 18:39:24.367507 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.367477    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zj4zw"
Sep 16 18:39:24.375510 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.375479    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tch8d"
Sep 16 18:39:24.380936 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.380908    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8dncf"
Sep 16 18:39:24.388970 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.388946    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hhl4g"
Sep 16 18:39:24.396820 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.396794    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:24.406073 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.406049    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8pj92"
Sep 16 18:39:24.411880 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.411856    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t9j4z"
Sep 16 18:39:24.419612 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.419587    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r596t"
Sep 16 18:39:24.677058 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.676975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs\") pod \"network-metrics-daemon-7r6xr\" (UID: \"2344014b-38cc-448f-9c9d-6b49d8e80b85\") " pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:24.677210 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:24.677138    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:24.677260 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:24.677213    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs podName:2344014b-38cc-448f-9c9d-6b49d8e80b85 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:25.677196042 +0000 UTC m=+4.036714016 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs") pod "network-metrics-daemon-7r6xr" (UID: "2344014b-38cc-448f-9c9d-6b49d8e80b85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:24.777900 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.777861    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqfsl\" (UniqueName: \"kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl\") pod \"network-check-target-hqh7p\" (UID: \"dbb3e980-5ba7-4b12-82b2-1fa036ec00b7\") " pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:24.777900 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:24.777908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret\") pod \"global-pull-secret-syncer-ddtnd\" (UID: \"cea41c21-51ad-46c1-8f7c-fc4dab1008e7\") " pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:24.778284 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:24.778024    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:24.778284 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:24.778033    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:24.778284 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:24.778055    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:24.778284 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:24.778065    2565 projected.go:196] Error preparing data for projected volume kube-api-access-xqfsl for pod openshift-network-diagnostics/network-check-target-hqh7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:24.778284 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:24.778084    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret podName:cea41c21-51ad-46c1-8f7c-fc4dab1008e7 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:25.778067608 +0000 UTC m=+4.137585586 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret") pod "global-pull-secret-syncer-ddtnd" (UID: "cea41c21-51ad-46c1-8f7c-fc4dab1008e7") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:24.778284 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:24.778107    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl podName:dbb3e980-5ba7-4b12-82b2-1fa036ec00b7 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:25.77809449 +0000 UTC m=+4.137612480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqfsl" (UniqueName: "kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl") pod "network-check-target-hqh7p" (UID: "dbb3e980-5ba7-4b12-82b2-1fa036ec00b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:25.113881 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:25.113756    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:34:24 +0000 UTC" deadline="2028-02-21 12:44:36.569720769 +0000 UTC"
Sep 16 18:39:25.113881 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:25.113823    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12546h5m11.455901865s"
Sep 16 18:39:25.162693 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:25.162595    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe676eb4_48c2_485b_9752_2198abad740a.slice/crio-ec328ba406bd440e027c698c8c5f48ffec72cc97b0340a9f84f8e3013c1127ea WatchSource:0}: Error finding container ec328ba406bd440e027c698c8c5f48ffec72cc97b0340a9f84f8e3013c1127ea: Status 404 returned error can't find the container with id ec328ba406bd440e027c698c8c5f48ffec72cc97b0340a9f84f8e3013c1127ea
Sep 16 18:39:25.164615 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:25.164550    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7872de0b_da8d_4589_afaa_8675716cd80b.slice/crio-a6eee6cf67c82cf907fe128ed04a0cdea010d9784d1320b926b82e71bfb817a2 WatchSource:0}: Error finding container a6eee6cf67c82cf907fe128ed04a0cdea010d9784d1320b926b82e71bfb817a2: Status 404 returned error can't find the container with id a6eee6cf67c82cf907fe128ed04a0cdea010d9784d1320b926b82e71bfb817a2
Sep 16 18:39:25.166124 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:25.166080    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef3366e_85c0_4283_a4f0_aceaf04445a4.slice/crio-61f892d9b829661a09a33a1424e0017a1c89e7243b9615ca49829679a3b8ec92 WatchSource:0}: Error finding container 61f892d9b829661a09a33a1424e0017a1c89e7243b9615ca49829679a3b8ec92: Status 404 returned error can't find the container with id 61f892d9b829661a09a33a1424e0017a1c89e7243b9615ca49829679a3b8ec92
Sep 16 18:39:25.169455 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:25.169417    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5368e2c9_26dd_429b_827a_270b3c2fe324.slice/crio-ae0dc08858f16aae1a53285f4e76a80f43f897b1122e799f5fa8d542bce750a4 WatchSource:0}: Error finding container ae0dc08858f16aae1a53285f4e76a80f43f897b1122e799f5fa8d542bce750a4: Status 404 returned error can't find the container with id ae0dc08858f16aae1a53285f4e76a80f43f897b1122e799f5fa8d542bce750a4
Sep 16 18:39:25.170636 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:25.170607    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd830733_5618_401e_945e_21a97bb4b76f.slice/crio-a625e29448d0f25a3eb775043a55a7cc2e92b09dc132d8822b8db9eaab453a5b WatchSource:0}: Error finding container a625e29448d0f25a3eb775043a55a7cc2e92b09dc132d8822b8db9eaab453a5b: Status 404 returned error can't find the container with id a625e29448d0f25a3eb775043a55a7cc2e92b09dc132d8822b8db9eaab453a5b
Sep 16 18:39:25.171449 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:25.171387    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf251ea42_1492_4bf4_942e_fdd522e5bbc5.slice/crio-2ac0caef1be1220c0b5265970bf3a678b4c8e6400afccd80a90bf239f33fbe05 WatchSource:0}: Error finding container 2ac0caef1be1220c0b5265970bf3a678b4c8e6400afccd80a90bf239f33fbe05: Status 404 returned error can't find the container with id 2ac0caef1be1220c0b5265970bf3a678b4c8e6400afccd80a90bf239f33fbe05
Sep 16 18:39:25.173924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:25.172928    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dce3890_5613_4464_91b5_388c8579cfee.slice/crio-44d2e9b63ca8a25e95f08fd7a8c34f1c33dd650d4134c5e1af1fbdb0355451c6 WatchSource:0}: Error finding container 44d2e9b63ca8a25e95f08fd7a8c34f1c33dd650d4134c5e1af1fbdb0355451c6: Status 404 returned error can't find the container with id 44d2e9b63ca8a25e95f08fd7a8c34f1c33dd650d4134c5e1af1fbdb0355451c6
Sep 16 18:39:25.174419 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:25.174088    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1cdba36_53e5_400a_9fdd_7c62a3fe0086.slice/crio-efb75c6125c4e78210090d79ab692b19c2360bfcb73ca5c7fb0a0347c35342d8 WatchSource:0}: Error finding container efb75c6125c4e78210090d79ab692b19c2360bfcb73ca5c7fb0a0347c35342d8: Status 404 returned error can't find the container with id efb75c6125c4e78210090d79ab692b19c2360bfcb73ca5c7fb0a0347c35342d8
Sep 16 18:39:25.174964 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:25.174835    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc940a913_b6ea_4d59_9e84_60a83842c3a1.slice/crio-a574c6663b2e3a82b205dc46987f64168fb06a829cf98901f5e5d04a1f2408c5 WatchSource:0}: Error finding container a574c6663b2e3a82b205dc46987f64168fb06a829cf98901f5e5d04a1f2408c5: Status 404 returned error can't find the container with id a574c6663b2e3a82b205dc46987f64168fb06a829cf98901f5e5d04a1f2408c5
Sep 16 18:39:25.201117 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:25.201091    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:25.201245 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:25.201198    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7r6xr" podUID="2344014b-38cc-448f-9c9d-6b49d8e80b85"
Sep 16 18:39:25.210847 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:25.210809    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8dncf" event={"ID":"2dce3890-5613-4464-91b5-388c8579cfee","Type":"ContainerStarted","Data":"44d2e9b63ca8a25e95f08fd7a8c34f1c33dd650d4134c5e1af1fbdb0355451c6"}
Sep 16 18:39:25.212260 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:25.212227    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t9j4z" event={"ID":"dd830733-5618-401e-945e-21a97bb4b76f","Type":"ContainerStarted","Data":"a625e29448d0f25a3eb775043a55a7cc2e92b09dc132d8822b8db9eaab453a5b"}
Sep 16 18:39:25.213389 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:25.213359    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r596t" event={"ID":"9ef3366e-85c0-4283-a4f0-aceaf04445a4","Type":"ContainerStarted","Data":"61f892d9b829661a09a33a1424e0017a1c89e7243b9615ca49829679a3b8ec92"}
Sep 16 18:39:25.217201 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:25.217170    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lpp26" event={"ID":"7872de0b-da8d-4589-afaa-8675716cd80b","Type":"ContainerStarted","Data":"a6eee6cf67c82cf907fe128ed04a0cdea010d9784d1320b926b82e71bfb817a2"}
Sep 16 18:39:25.219390 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:25.219365    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb" event={"ID":"c940a913-b6ea-4d59-9e84-60a83842c3a1","Type":"ContainerStarted","Data":"a574c6663b2e3a82b205dc46987f64168fb06a829cf98901f5e5d04a1f2408c5"}
Sep 16 18:39:25.220445 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:25.220425    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hhl4g" event={"ID":"f251ea42-1492-4bf4-942e-fdd522e5bbc5","Type":"ContainerStarted","Data":"2ac0caef1be1220c0b5265970bf3a678b4c8e6400afccd80a90bf239f33fbe05"}
Sep 16 18:39:25.222139 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:25.222110    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zj4zw" event={"ID":"5368e2c9-26dd-429b-827a-270b3c2fe324","Type":"ContainerStarted","Data":"ae0dc08858f16aae1a53285f4e76a80f43f897b1122e799f5fa8d542bce750a4"}
Sep 16 18:39:25.223223 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:25.223196    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tch8d" event={"ID":"fe676eb4-48c2-485b-9752-2198abad740a","Type":"ContainerStarted","Data":"ec328ba406bd440e027c698c8c5f48ffec72cc97b0340a9f84f8e3013c1127ea"}
Sep 16 18:39:25.224302 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:25.224280    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8pj92" event={"ID":"d1cdba36-53e5-400a-9fdd-7c62a3fe0086","Type":"ContainerStarted","Data":"efb75c6125c4e78210090d79ab692b19c2360bfcb73ca5c7fb0a0347c35342d8"}
Sep 16 18:39:25.684965 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:25.684273    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs\") pod \"network-metrics-daemon-7r6xr\" (UID: \"2344014b-38cc-448f-9c9d-6b49d8e80b85\") " pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:25.684965 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:25.684476    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:25.684965 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:25.684548    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs podName:2344014b-38cc-448f-9c9d-6b49d8e80b85 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:27.684527374 +0000 UTC m=+6.044045358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs") pod "network-metrics-daemon-7r6xr" (UID: "2344014b-38cc-448f-9c9d-6b49d8e80b85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:25.785504 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:25.785458    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqfsl\" (UniqueName: \"kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl\") pod \"network-check-target-hqh7p\" (UID: \"dbb3e980-5ba7-4b12-82b2-1fa036ec00b7\") " pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:25.785703 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:25.785534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret\") pod \"global-pull-secret-syncer-ddtnd\" (UID: \"cea41c21-51ad-46c1-8f7c-fc4dab1008e7\") " pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:25.785703 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:25.785691    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:25.785818 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:25.785760    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret podName:cea41c21-51ad-46c1-8f7c-fc4dab1008e7 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:27.785740139 +0000 UTC m=+6.145258116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret") pod "global-pull-secret-syncer-ddtnd" (UID: "cea41c21-51ad-46c1-8f7c-fc4dab1008e7") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:25.786579 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:25.786552    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:25.786676 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:25.786588    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:25.786676 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:25.786607    2565 projected.go:196] Error preparing data for projected volume kube-api-access-xqfsl for pod openshift-network-diagnostics/network-check-target-hqh7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:25.786797 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:25.786682    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl podName:dbb3e980-5ba7-4b12-82b2-1fa036ec00b7 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:27.786665048 +0000 UTC m=+6.146183026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqfsl" (UniqueName: "kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl") pod "network-check-target-hqh7p" (UID: "dbb3e980-5ba7-4b12-82b2-1fa036ec00b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:26.114902 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:26.114518    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:34:24 +0000 UTC" deadline="2028-04-16 18:57:07.219089396 +0000 UTC"
Sep 16 18:39:26.114902 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:26.114819    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13872h17m41.104274915s"
Sep 16 18:39:26.203937 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:26.203902    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:26.204241 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:26.204043    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddtnd" podUID="cea41c21-51ad-46c1-8f7c-fc4dab1008e7"
Sep 16 18:39:26.204489 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:26.204467    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:26.204592 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:26.204569    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqh7p" podUID="dbb3e980-5ba7-4b12-82b2-1fa036ec00b7"
Sep 16 18:39:26.242827 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:26.242790    2565 generic.go:356] "Generic (PLEG): container finished" podID="832091a938dd3fb43132057170181abf" containerID="2a5c0ff4ee4f7f7e3ad4b662519ad76ec95dfe3eae626cd6ca60a7be3ddc3050" exitCode=0
Sep 16 18:39:26.243833 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:26.243398    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal" event={"ID":"832091a938dd3fb43132057170181abf","Type":"ContainerDied","Data":"2a5c0ff4ee4f7f7e3ad4b662519ad76ec95dfe3eae626cd6ca60a7be3ddc3050"}
Sep 16 18:39:26.254650 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:26.253812    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-204.ec2.internal" event={"ID":"1152677edddc8ea58737be1eaabf066e","Type":"ContainerStarted","Data":"ee0fd47ba2fe9c22505430ecc7aba58e1a2b13e57f4198ccbeb33c2f88b14554"}
Sep 16 18:39:27.200876 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:27.200784    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:27.201419 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:27.200940    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7r6xr" podUID="2344014b-38cc-448f-9c9d-6b49d8e80b85"
Sep 16 18:39:27.274451 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:27.274411    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal" event={"ID":"832091a938dd3fb43132057170181abf","Type":"ContainerStarted","Data":"42bbf5b0f6a59b7dff0cc63c2dfc09edc7dc2fd5146677471c61b3ba88db0208"}
Sep 16 18:39:27.293806 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:27.293737    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-204.ec2.internal" podStartSLOduration=4.293713253 podStartE2EDuration="4.293713253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:26.291802992 +0000 UTC m=+4.651320988" watchObservedRunningTime="2026-09-16 18:39:27.293713253 +0000 UTC m=+5.653231250"
Sep 16 18:39:27.294174 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:27.294007    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-204.ec2.internal" podStartSLOduration=4.293992371 podStartE2EDuration="4.293992371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:27.292789068 +0000 UTC m=+5.652307065" watchObservedRunningTime="2026-09-16 18:39:27.293992371 +0000 UTC m=+5.653510367"
Sep 16 18:39:27.702592 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:27.702545    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs\") pod \"network-metrics-daemon-7r6xr\" (UID: \"2344014b-38cc-448f-9c9d-6b49d8e80b85\") " pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:27.702886 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:27.702708    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:27.702886 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:27.702776    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs podName:2344014b-38cc-448f-9c9d-6b49d8e80b85 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:31.702755999 +0000 UTC m=+10.062273986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs") pod "network-metrics-daemon-7r6xr" (UID: "2344014b-38cc-448f-9c9d-6b49d8e80b85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:27.803408 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:27.803367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqfsl\" (UniqueName: \"kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl\") pod \"network-check-target-hqh7p\" (UID: \"dbb3e980-5ba7-4b12-82b2-1fa036ec00b7\") " pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:27.803611 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:27.803434    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret\") pod \"global-pull-secret-syncer-ddtnd\" (UID: \"cea41c21-51ad-46c1-8f7c-fc4dab1008e7\") " pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:27.803611 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:27.803592    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:27.803737 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:27.803656    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret podName:cea41c21-51ad-46c1-8f7c-fc4dab1008e7 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:31.803637091 +0000 UTC m=+10.163155071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret") pod "global-pull-secret-syncer-ddtnd" (UID: "cea41c21-51ad-46c1-8f7c-fc4dab1008e7") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:27.804087 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:27.804065    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:27.804184 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:27.804093    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:27.804184 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:27.804107    2565 projected.go:196] Error preparing data for projected volume kube-api-access-xqfsl for pod openshift-network-diagnostics/network-check-target-hqh7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:27.804184 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:27.804150    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl podName:dbb3e980-5ba7-4b12-82b2-1fa036ec00b7 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:31.804135715 +0000 UTC m=+10.163653688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqfsl" (UniqueName: "kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl") pod "network-check-target-hqh7p" (UID: "dbb3e980-5ba7-4b12-82b2-1fa036ec00b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:28.201838 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:28.201752    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:28.201838 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:28.201782    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:28.202362 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:28.201906    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqh7p" podUID="dbb3e980-5ba7-4b12-82b2-1fa036ec00b7"
Sep 16 18:39:28.202362 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:28.202024    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddtnd" podUID="cea41c21-51ad-46c1-8f7c-fc4dab1008e7"
Sep 16 18:39:29.200891 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:29.200851    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:29.201065 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:29.201004    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7r6xr" podUID="2344014b-38cc-448f-9c9d-6b49d8e80b85"
Sep 16 18:39:30.203183 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:30.202467    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:30.203183 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:30.202605    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddtnd" podUID="cea41c21-51ad-46c1-8f7c-fc4dab1008e7"
Sep 16 18:39:30.203183 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:30.203038    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:30.203183 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:30.203138    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqh7p" podUID="dbb3e980-5ba7-4b12-82b2-1fa036ec00b7"
Sep 16 18:39:31.201108 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:31.201072    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:31.201303 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:31.201221    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7r6xr" podUID="2344014b-38cc-448f-9c9d-6b49d8e80b85"
Sep 16 18:39:31.735519 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:31.735470    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs\") pod \"network-metrics-daemon-7r6xr\" (UID: \"2344014b-38cc-448f-9c9d-6b49d8e80b85\") " pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:31.735981 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:31.735656    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:31.735981 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:31.735723    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs podName:2344014b-38cc-448f-9c9d-6b49d8e80b85 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:39.735704059 +0000 UTC m=+18.095222037 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs") pod "network-metrics-daemon-7r6xr" (UID: "2344014b-38cc-448f-9c9d-6b49d8e80b85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:31.836882 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:31.836839    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqfsl\" (UniqueName: \"kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl\") pod \"network-check-target-hqh7p\" (UID: \"dbb3e980-5ba7-4b12-82b2-1fa036ec00b7\") " pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:31.836882 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:31.836903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret\") pod \"global-pull-secret-syncer-ddtnd\" (UID: \"cea41c21-51ad-46c1-8f7c-fc4dab1008e7\") " pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:31.837185 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:31.837046    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:31.837185 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:31.837053    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:31.837185 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:31.837086    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:31.837185 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:31.837101    2565 projected.go:196] Error preparing data for projected volume kube-api-access-xqfsl for pod openshift-network-diagnostics/network-check-target-hqh7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:31.837185 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:31.837109    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret podName:cea41c21-51ad-46c1-8f7c-fc4dab1008e7 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:39.837090526 +0000 UTC m=+18.196608505 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret") pod "global-pull-secret-syncer-ddtnd" (UID: "cea41c21-51ad-46c1-8f7c-fc4dab1008e7") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:31.837185 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:31.837156    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl podName:dbb3e980-5ba7-4b12-82b2-1fa036ec00b7 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:39.837138511 +0000 UTC m=+18.196656488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqfsl" (UniqueName: "kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl") pod "network-check-target-hqh7p" (UID: "dbb3e980-5ba7-4b12-82b2-1fa036ec00b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:32.202929 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:32.202386    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:32.202929 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:32.202506    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddtnd" podUID="cea41c21-51ad-46c1-8f7c-fc4dab1008e7"
Sep 16 18:39:32.202929 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:32.202834    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:32.203202 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:32.203007    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqh7p" podUID="dbb3e980-5ba7-4b12-82b2-1fa036ec00b7"
Sep 16 18:39:33.201461 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:33.201422    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:33.201913 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:33.201575    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7r6xr" podUID="2344014b-38cc-448f-9c9d-6b49d8e80b85"
Sep 16 18:39:34.200908 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:34.200874    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:34.201096 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:34.200918    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:34.201096 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:34.201001    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddtnd" podUID="cea41c21-51ad-46c1-8f7c-fc4dab1008e7"
Sep 16 18:39:34.201202 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:34.201112    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqh7p" podUID="dbb3e980-5ba7-4b12-82b2-1fa036ec00b7"
Sep 16 18:39:35.201165 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:35.201127    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:35.201647 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:35.201268    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7r6xr" podUID="2344014b-38cc-448f-9c9d-6b49d8e80b85"
Sep 16 18:39:36.201097 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:36.201060    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:36.201312 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:36.201062    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:36.201312 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:36.201199    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqh7p" podUID="dbb3e980-5ba7-4b12-82b2-1fa036ec00b7"
Sep 16 18:39:36.201312 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:36.201281    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddtnd" podUID="cea41c21-51ad-46c1-8f7c-fc4dab1008e7"
Sep 16 18:39:37.200901 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:37.200857    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:37.201096 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:37.201020    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7r6xr" podUID="2344014b-38cc-448f-9c9d-6b49d8e80b85"
Sep 16 18:39:38.200940 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:38.200894    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:38.201444 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:38.200894    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:38.201444 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:38.201040    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddtnd" podUID="cea41c21-51ad-46c1-8f7c-fc4dab1008e7"
Sep 16 18:39:38.201444 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:38.201114    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqh7p" podUID="dbb3e980-5ba7-4b12-82b2-1fa036ec00b7"
Sep 16 18:39:39.201322 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:39.201282    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:39.201746 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:39.201424    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7r6xr" podUID="2344014b-38cc-448f-9c9d-6b49d8e80b85"
Sep 16 18:39:39.802939 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:39.802894    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs\") pod \"network-metrics-daemon-7r6xr\" (UID: \"2344014b-38cc-448f-9c9d-6b49d8e80b85\") " pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:39.803130 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:39.803103    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:39.803190 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:39.803182    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs podName:2344014b-38cc-448f-9c9d-6b49d8e80b85 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:55.803160298 +0000 UTC m=+34.162678275 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs") pod "network-metrics-daemon-7r6xr" (UID: "2344014b-38cc-448f-9c9d-6b49d8e80b85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:39.903984 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:39.903941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqfsl\" (UniqueName: \"kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl\") pod \"network-check-target-hqh7p\" (UID: \"dbb3e980-5ba7-4b12-82b2-1fa036ec00b7\") " pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:39.904179 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:39.904008    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret\") pod \"global-pull-secret-syncer-ddtnd\" (UID: \"cea41c21-51ad-46c1-8f7c-fc4dab1008e7\") " pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:39.904179 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:39.904136    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:39.904179 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:39.904176    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:39.904314 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:39.904188    2565 projected.go:196] Error preparing data for projected volume kube-api-access-xqfsl for pod openshift-network-diagnostics/network-check-target-hqh7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:39.904314 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:39.904149    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:39.904314 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:39.904250    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl podName:dbb3e980-5ba7-4b12-82b2-1fa036ec00b7 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:55.90422996 +0000 UTC m=+34.263747934 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqfsl" (UniqueName: "kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl") pod "network-check-target-hqh7p" (UID: "dbb3e980-5ba7-4b12-82b2-1fa036ec00b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:39.904489 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:39.904313    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret podName:cea41c21-51ad-46c1-8f7c-fc4dab1008e7 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:55.904284843 +0000 UTC m=+34.263802817 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret") pod "global-pull-secret-syncer-ddtnd" (UID: "cea41c21-51ad-46c1-8f7c-fc4dab1008e7") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:40.201763 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:40.201626    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:40.201763 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:40.201639    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:40.202333 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:40.201773    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddtnd" podUID="cea41c21-51ad-46c1-8f7c-fc4dab1008e7"
Sep 16 18:39:40.202333 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:40.201844    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqh7p" podUID="dbb3e980-5ba7-4b12-82b2-1fa036ec00b7"
Sep 16 18:39:41.201306 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:41.201265    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:41.201504 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:41.201405    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7r6xr" podUID="2344014b-38cc-448f-9c9d-6b49d8e80b85"
Sep 16 18:39:42.201988 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:42.201950    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:42.202437 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:42.202075    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:42.202437 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:42.202088    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqh7p" podUID="dbb3e980-5ba7-4b12-82b2-1fa036ec00b7"
Sep 16 18:39:42.202437 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:42.202184    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddtnd" podUID="cea41c21-51ad-46c1-8f7c-fc4dab1008e7"
Sep 16 18:39:43.201457 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:43.201434    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:43.201568 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:43.201550    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7r6xr" podUID="2344014b-38cc-448f-9c9d-6b49d8e80b85"
Sep 16 18:39:44.202032 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.201577    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:44.202887 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.201631    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:44.202887 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:44.202158    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddtnd" podUID="cea41c21-51ad-46c1-8f7c-fc4dab1008e7"
Sep 16 18:39:44.202887 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:44.202221    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqh7p" podUID="dbb3e980-5ba7-4b12-82b2-1fa036ec00b7"
Sep 16 18:39:44.312082 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.312032    2565 generic.go:356] "Generic (PLEG): container finished" podID="f251ea42-1492-4bf4-942e-fdd522e5bbc5" containerID="ab66b7340e7d5734be97bd1f6aa3f7fcccff981cd7db47400e9b6dd4bdc4791f" exitCode=0
Sep 16 18:39:44.312290 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.312117    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hhl4g" event={"ID":"f251ea42-1492-4bf4-942e-fdd522e5bbc5","Type":"ContainerDied","Data":"ab66b7340e7d5734be97bd1f6aa3f7fcccff981cd7db47400e9b6dd4bdc4791f"}
Sep 16 18:39:44.314042 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.314013    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zj4zw" event={"ID":"5368e2c9-26dd-429b-827a-270b3c2fe324","Type":"ContainerStarted","Data":"30ba1f6d3941d1d9324e43f57470d6cf4145b57614beb2054bf24102acff4020"}
Sep 16 18:39:44.315701 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.315679    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tch8d" event={"ID":"fe676eb4-48c2-485b-9752-2198abad740a","Type":"ContainerStarted","Data":"866f9628993c7586de9466d6798f13f2731a221cc4d2a7110af247cd70942a5e"}
Sep 16 18:39:44.317601 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.317573    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8pj92" event={"ID":"d1cdba36-53e5-400a-9fdd-7c62a3fe0086","Type":"ContainerStarted","Data":"40792b8bc342c27e0d2860749b422f04eade168a59c8b1295173ce2692a189ce"}
Sep 16 18:39:44.319054 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.319032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8dncf" event={"ID":"2dce3890-5613-4464-91b5-388c8579cfee","Type":"ContainerStarted","Data":"f2a397fb30b497cf31dd58cdf625c7c4f7dbf5bd6e976eb498d75486216ef2db"}
Sep 16 18:39:44.320456 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.320433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t9j4z" event={"ID":"dd830733-5618-401e-945e-21a97bb4b76f","Type":"ContainerStarted","Data":"310ed07854d68ea74ae425ff1e4d4fa944e56d12958fcd1ce02919d4877dc0b0"}
Sep 16 18:39:44.323401 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.323338    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lpp26" event={"ID":"7872de0b-da8d-4589-afaa-8675716cd80b","Type":"ContainerStarted","Data":"45d3fd114900cc3a3e4c94cf4dacb942f756862d0d94c3ff3096bf7bfe29a619"}
Sep 16 18:39:44.323495 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.323410    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lpp26" event={"ID":"7872de0b-da8d-4589-afaa-8675716cd80b","Type":"ContainerStarted","Data":"6bb4367d3906ca1f9aafdc79b3a6b446efa6dfdf5cf7a5c18723b5ae316693a1"}
Sep 16 18:39:44.323495 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.323447    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lpp26" event={"ID":"7872de0b-da8d-4589-afaa-8675716cd80b","Type":"ContainerStarted","Data":"e1dd78ee92545aee2d6dae69ce5589b6e8bc8e4331af698b7c39c9c120b6c9b6"}
Sep 16 18:39:44.323495 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.323477    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lpp26" event={"ID":"7872de0b-da8d-4589-afaa-8675716cd80b","Type":"ContainerStarted","Data":"e9a70f160a200cf2c0cbc85113b5e5cac71ae9ff8b6381e4751b2f23e6cf4c8a"}
Sep 16 18:39:44.323592 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.323505    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lpp26" event={"ID":"7872de0b-da8d-4589-afaa-8675716cd80b","Type":"ContainerStarted","Data":"9b488789427a040d4932ee7a6c2f25298d0a523e987a749692258cb0da0fefb7"}
Sep 16 18:39:44.323592 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.323537    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lpp26" event={"ID":"7872de0b-da8d-4589-afaa-8675716cd80b","Type":"ContainerStarted","Data":"d0fe2b5b9f575c463013f65c93a3cc81e2f2d879e5b42f0d624d9ff2538a5ba5"}
Sep 16 18:39:44.324700 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.324678    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb" event={"ID":"c940a913-b6ea-4d59-9e84-60a83842c3a1","Type":"ContainerStarted","Data":"260902a8f0a9c21fcab70cfed20afc56fcb286751d19f8cc9a1532503e4ee83d"}
Sep 16 18:39:44.355950 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.355892    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tch8d" podStartSLOduration=4.311883439 podStartE2EDuration="22.355875559s" totalImagesPullingTime="18.04399212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:44.355113268 +0000 UTC m=+22.714631262" watchObservedRunningTime="2026-09-16 18:39:44.355875559 +0000 UTC m=+22.715393554"
Sep 16 18:39:44.412892 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.411563    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zj4zw" podStartSLOduration=4.343570583 podStartE2EDuration="22.411541723s" totalImagesPullingTime="18.06797114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:44.410633729 +0000 UTC m=+22.770151723" watchObservedRunningTime="2026-09-16 18:39:44.411541723 +0000 UTC m=+22.771059719"
Sep 16 18:39:44.412892 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.411802    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8dncf" podStartSLOduration=4.07817184 podStartE2EDuration="22.411794731s" totalImagesPullingTime="18.333622891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:44.377083561 +0000 UTC m=+22.736601555" watchObservedRunningTime="2026-09-16 18:39:44.411794731 +0000 UTC m=+22.771312727"
Sep 16 18:39:44.471785 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.471734    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t9j4z" podStartSLOduration=8.630870539 podStartE2EDuration="22.47171634s" totalImagesPullingTime="13.840845801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:44.471368327 +0000 UTC m=+22.830886317" watchObservedRunningTime="2026-09-16 18:39:44.47171634 +0000 UTC m=+22.831234336"
Sep 16 18:39:44.471992 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.471901    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8pj92" podStartSLOduration=4.439117333 podStartE2EDuration="22.471894054s" totalImagesPullingTime="18.032776721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:44.438633097 +0000 UTC m=+22.798151091" watchObservedRunningTime="2026-09-16 18:39:44.471894054 +0000 UTC m=+22.831412048"
Sep 16 18:39:44.808959 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:44.808929    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:39:45.200989 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:45.200951    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:45.201172 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:45.201112    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7r6xr" podUID="2344014b-38cc-448f-9c9d-6b49d8e80b85"
Sep 16 18:39:45.259638 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:45.259504    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:39:44.808950812Z","UUID":"85cf748e-2a25-4a1e-8f64-2ed4e6799ce5","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:39:45.261253 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:45.261223    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:39:45.261408 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:45.261266    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:39:45.331392 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:45.331277    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb" event={"ID":"c940a913-b6ea-4d59-9e84-60a83842c3a1","Type":"ContainerStarted","Data":"8dccdd479f2eb90634bf60e6e38348fc1793c790f63ca489420f14816505ae9c"}
Sep 16 18:39:45.334602 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:45.334473    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r596t" event={"ID":"9ef3366e-85c0-4283-a4f0-aceaf04445a4","Type":"ContainerStarted","Data":"7720a5cd14d1daa58e0dded60c48d99dd96673591958778b1b03db4673522d8e"}
Sep 16 18:39:45.351603 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:45.351538    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r596t" podStartSLOduration=5.281519044 podStartE2EDuration="23.351513084s" totalImagesPullingTime="18.06999404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:45.35098022 +0000 UTC m=+23.710498217" watchObservedRunningTime="2026-09-16 18:39:45.351513084 +0000 UTC m=+23.711031078"
Sep 16 18:39:45.948578 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:45.948544    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t9j4z"
Sep 16 18:39:45.949187 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:45.949165    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t9j4z"
Sep 16 18:39:46.201318 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:46.201284    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:46.201506 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:46.201284    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:46.201506 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:46.201431    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddtnd" podUID="cea41c21-51ad-46c1-8f7c-fc4dab1008e7"
Sep 16 18:39:46.201506 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:46.201451    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqh7p" podUID="dbb3e980-5ba7-4b12-82b2-1fa036ec00b7"
Sep 16 18:39:46.341060 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:46.340965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lpp26" event={"ID":"7872de0b-da8d-4589-afaa-8675716cd80b","Type":"ContainerStarted","Data":"633b9fbd20252bd29cf6f89de635bc8a6ea282ef4c04a822439de9a0b1cef8a0"}
Sep 16 18:39:46.343587 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:46.343542    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb" event={"ID":"c940a913-b6ea-4d59-9e84-60a83842c3a1","Type":"ContainerStarted","Data":"5fe0ad3c02644f7872140c1f2d55688e673abaa970b0dddb042b5ad8c665a863"}
Sep 16 18:39:46.344107 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:46.344086    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t9j4z"
Sep 16 18:39:46.344528 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:46.344510    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t9j4z"
Sep 16 18:39:46.425797 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:46.425735    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-skqkb" podStartSLOduration=4.2726534560000005 podStartE2EDuration="24.425716093s" totalImagesPullingTime="20.153062637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:46.389853545 +0000 UTC m=+24.749371541" watchObservedRunningTime="2026-09-16 18:39:46.425716093 +0000 UTC m=+24.785234088"
Sep 16 18:39:47.201169 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:47.201132    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:47.201380 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:47.201275    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7r6xr" podUID="2344014b-38cc-448f-9c9d-6b49d8e80b85"
Sep 16 18:39:48.201536 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:48.201494    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:48.201955 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:48.201634    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqh7p" podUID="dbb3e980-5ba7-4b12-82b2-1fa036ec00b7"
Sep 16 18:39:48.201955 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:48.201497    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:48.201955 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:48.201748    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddtnd" podUID="cea41c21-51ad-46c1-8f7c-fc4dab1008e7"
Sep 16 18:39:49.201637 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:49.201433    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:49.202511 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:49.201757    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7r6xr" podUID="2344014b-38cc-448f-9c9d-6b49d8e80b85"
Sep 16 18:39:49.355107 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:49.355069    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lpp26" event={"ID":"7872de0b-da8d-4589-afaa-8675716cd80b","Type":"ContainerStarted","Data":"9653ab48587f7227ec8cbbf633350c4fb4b2ef165466e23d7f716bbebcd74341"}
Sep 16 18:39:49.355311 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:49.355292    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:49.355414 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:49.355370    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:49.356957 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:49.356934    2565 generic.go:356] "Generic (PLEG): container finished" podID="f251ea42-1492-4bf4-942e-fdd522e5bbc5" containerID="ac39113f3cf240e046c055e1acdd62080d48e6aa00b4a605487ce94b458dbd67" exitCode=0
Sep 16 18:39:49.357081 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:49.356967    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hhl4g" event={"ID":"f251ea42-1492-4bf4-942e-fdd522e5bbc5","Type":"ContainerDied","Data":"ac39113f3cf240e046c055e1acdd62080d48e6aa00b4a605487ce94b458dbd67"}
Sep 16 18:39:49.371542 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:49.371515    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:49.430866 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:49.430810    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lpp26" podStartSLOduration=9.354862899 podStartE2EDuration="27.430794016s" totalImagesPullingTime="18.075931117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:49.393047876 +0000 UTC m=+27.752565864" watchObservedRunningTime="2026-09-16 18:39:49.430794016 +0000 UTC m=+27.790311992"
Sep 16 18:39:50.201899 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:50.201631    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:50.202522 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:50.202007    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddtnd" podUID="cea41c21-51ad-46c1-8f7c-fc4dab1008e7"
Sep 16 18:39:50.202522 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:50.201674    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:50.202522 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:50.202117    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqh7p" podUID="dbb3e980-5ba7-4b12-82b2-1fa036ec00b7"
Sep 16 18:39:50.362331 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:50.362295    2565 generic.go:356] "Generic (PLEG): container finished" podID="f251ea42-1492-4bf4-942e-fdd522e5bbc5" containerID="555d9dfa022ea30cb4eca343f6d6f9dd5141711059da07763a78e1ae07db62ac" exitCode=0
Sep 16 18:39:50.362515 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:50.362380    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hhl4g" event={"ID":"f251ea42-1492-4bf4-942e-fdd522e5bbc5","Type":"ContainerDied","Data":"555d9dfa022ea30cb4eca343f6d6f9dd5141711059da07763a78e1ae07db62ac"}
Sep 16 18:39:50.362953 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:50.362847    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:50.379395 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:50.379272    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:39:50.684010 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:50.683920    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ddtnd"]
Sep 16 18:39:50.684165 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:50.684038    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:50.684165 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:50.684139    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddtnd" podUID="cea41c21-51ad-46c1-8f7c-fc4dab1008e7"
Sep 16 18:39:50.688973 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:50.688927    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hqh7p"]
Sep 16 18:39:50.689129 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:50.689036    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:50.689205 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:50.689155    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqh7p" podUID="dbb3e980-5ba7-4b12-82b2-1fa036ec00b7"
Sep 16 18:39:50.692214 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:50.690497    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7r6xr"]
Sep 16 18:39:50.692214 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:50.690671    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:50.692214 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:50.690810    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7r6xr" podUID="2344014b-38cc-448f-9c9d-6b49d8e80b85"
Sep 16 18:39:52.201433 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:52.201408    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:52.201795 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:52.201503    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7r6xr" podUID="2344014b-38cc-448f-9c9d-6b49d8e80b85"
Sep 16 18:39:52.201795 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:52.201599    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:52.201795 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:52.201706    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqh7p" podUID="dbb3e980-5ba7-4b12-82b2-1fa036ec00b7"
Sep 16 18:39:52.201795 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:52.201738    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:52.201939 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:52.201815    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddtnd" podUID="cea41c21-51ad-46c1-8f7c-fc4dab1008e7"
Sep 16 18:39:52.369579 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:52.369541    2565 generic.go:356] "Generic (PLEG): container finished" podID="f251ea42-1492-4bf4-942e-fdd522e5bbc5" containerID="953a8b4222050ee73d155893e700b9289d2b1d5993685a9dcfca2d0944260c35" exitCode=0
Sep 16 18:39:52.369752 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:52.369623    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hhl4g" event={"ID":"f251ea42-1492-4bf4-942e-fdd522e5bbc5","Type":"ContainerDied","Data":"953a8b4222050ee73d155893e700b9289d2b1d5993685a9dcfca2d0944260c35"}
Sep 16 18:39:54.201441 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:54.201201    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:54.201917 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:54.201201    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:54.201917 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:54.201540    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7r6xr" podUID="2344014b-38cc-448f-9c9d-6b49d8e80b85"
Sep 16 18:39:54.201917 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:54.201337    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:54.201917 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:54.201601    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddtnd" podUID="cea41c21-51ad-46c1-8f7c-fc4dab1008e7"
Sep 16 18:39:54.201917 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:54.201666    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqh7p" podUID="dbb3e980-5ba7-4b12-82b2-1fa036ec00b7"
Sep 16 18:39:55.823741 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:55.823705    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs\") pod \"network-metrics-daemon-7r6xr\" (UID: \"2344014b-38cc-448f-9c9d-6b49d8e80b85\") " pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:55.824270 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:55.823890    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:55.824270 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:55.823961    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs podName:2344014b-38cc-448f-9c9d-6b49d8e80b85 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:27.823944124 +0000 UTC m=+66.183462097 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs") pod "network-metrics-daemon-7r6xr" (UID: "2344014b-38cc-448f-9c9d-6b49d8e80b85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:55.924141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:55.924094    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqfsl\" (UniqueName: \"kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl\") pod \"network-check-target-hqh7p\" (UID: \"dbb3e980-5ba7-4b12-82b2-1fa036ec00b7\") " pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:55.924373 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:55.924157    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret\") pod \"global-pull-secret-syncer-ddtnd\" (UID: \"cea41c21-51ad-46c1-8f7c-fc4dab1008e7\") " pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:55.924373 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:55.924298    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:55.924373 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:55.924337    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:55.924373 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:55.924304    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:55.924609 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:55.924450    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret podName:cea41c21-51ad-46c1-8f7c-fc4dab1008e7 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:27.924425835 +0000 UTC m=+66.283943833 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret") pod "global-pull-secret-syncer-ddtnd" (UID: "cea41c21-51ad-46c1-8f7c-fc4dab1008e7") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:55.924609 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:55.924368    2565 projected.go:196] Error preparing data for projected volume kube-api-access-xqfsl for pod openshift-network-diagnostics/network-check-target-hqh7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:55.924609 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:55.924503    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl podName:dbb3e980-5ba7-4b12-82b2-1fa036ec00b7 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:27.924491337 +0000 UTC m=+66.284009324 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqfsl" (UniqueName: "kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl") pod "network-check-target-hqh7p" (UID: "dbb3e980-5ba7-4b12-82b2-1fa036ec00b7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:56.204903 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.204871    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:56.204903 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.204902    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:56.205115 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.204964    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:56.205115 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:56.204978    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddtnd" podUID="cea41c21-51ad-46c1-8f7c-fc4dab1008e7"
Sep 16 18:39:56.205115 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:56.205056    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hqh7p" podUID="dbb3e980-5ba7-4b12-82b2-1fa036ec00b7"
Sep 16 18:39:56.205229 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:56.205155    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7r6xr" podUID="2344014b-38cc-448f-9c9d-6b49d8e80b85"
Sep 16 18:39:56.473113 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.473026    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-204.ec2.internal" event="NodeReady"
Sep 16 18:39:56.473283 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.473187    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:39:56.541746 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.541714    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x2brk"]
Sep 16 18:39:56.565077 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.565035    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vb8t5"]
Sep 16 18:39:56.565413 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.565182    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x2brk"
Sep 16 18:39:56.569394 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.569366    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vxh4z\""
Sep 16 18:39:56.569545 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.569366    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:39:56.569618 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.569542    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:39:56.586195 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.586153    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vb8t5"
Sep 16 18:39:56.586417 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.586160    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x2brk"]
Sep 16 18:39:56.588525 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.588498    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vb8t5"]
Sep 16 18:39:56.590194 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.590170    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:39:56.590307 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.590270    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kf94w\""
Sep 16 18:39:56.590819 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.590801    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:39:56.590916 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.590799    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:39:56.629195 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.629157    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls\") pod \"dns-default-x2brk\" (UID: \"359cd30d-7981-4184-8a6f-f63a2f585517\") " pod="openshift-dns/dns-default-x2brk"
Sep 16 18:39:56.629195 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.629192    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/359cd30d-7981-4184-8a6f-f63a2f585517-tmp-dir\") pod \"dns-default-x2brk\" (UID: \"359cd30d-7981-4184-8a6f-f63a2f585517\") " pod="openshift-dns/dns-default-x2brk"
Sep 16 18:39:56.629442 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.629250    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9mdb\" (UniqueName: \"kubernetes.io/projected/359cd30d-7981-4184-8a6f-f63a2f585517-kube-api-access-h9mdb\") pod \"dns-default-x2brk\" (UID: \"359cd30d-7981-4184-8a6f-f63a2f585517\") " pod="openshift-dns/dns-default-x2brk"
Sep 16 18:39:56.629442 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.629289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/359cd30d-7981-4184-8a6f-f63a2f585517-config-volume\") pod \"dns-default-x2brk\" (UID: \"359cd30d-7981-4184-8a6f-f63a2f585517\") " pod="openshift-dns/dns-default-x2brk"
Sep 16 18:39:56.729665 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.729579    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9mdb\" (UniqueName: \"kubernetes.io/projected/359cd30d-7981-4184-8a6f-f63a2f585517-kube-api-access-h9mdb\") pod \"dns-default-x2brk\" (UID: \"359cd30d-7981-4184-8a6f-f63a2f585517\") " pod="openshift-dns/dns-default-x2brk"
Sep 16 18:39:56.729665 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.729618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/359cd30d-7981-4184-8a6f-f63a2f585517-config-volume\") pod \"dns-default-x2brk\" (UID: \"359cd30d-7981-4184-8a6f-f63a2f585517\") " pod="openshift-dns/dns-default-x2brk"
Sep 16 18:39:56.729665 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.729639    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a00e71a-7efb-45c5-ac52-6e175ea6837a-cert\") pod \"ingress-canary-vb8t5\" (UID: \"1a00e71a-7efb-45c5-ac52-6e175ea6837a\") " pod="openshift-ingress-canary/ingress-canary-vb8t5"
Sep 16 18:39:56.729665 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.729660    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvlhx\" (UniqueName: \"kubernetes.io/projected/1a00e71a-7efb-45c5-ac52-6e175ea6837a-kube-api-access-kvlhx\") pod \"ingress-canary-vb8t5\" (UID: \"1a00e71a-7efb-45c5-ac52-6e175ea6837a\") " pod="openshift-ingress-canary/ingress-canary-vb8t5"
Sep 16 18:39:56.729964 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.729742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls\") pod \"dns-default-x2brk\" (UID: \"359cd30d-7981-4184-8a6f-f63a2f585517\") " pod="openshift-dns/dns-default-x2brk"
Sep 16 18:39:56.729964 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.729766    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/359cd30d-7981-4184-8a6f-f63a2f585517-tmp-dir\") pod \"dns-default-x2brk\" (UID: \"359cd30d-7981-4184-8a6f-f63a2f585517\") " pod="openshift-dns/dns-default-x2brk"
Sep 16 18:39:56.729964 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:56.729839    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:39:56.729964 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:56.729898    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls podName:359cd30d-7981-4184-8a6f-f63a2f585517 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:57.229878915 +0000 UTC m=+35.589396903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls") pod "dns-default-x2brk" (UID: "359cd30d-7981-4184-8a6f-f63a2f585517") : secret "dns-default-metrics-tls" not found
Sep 16 18:39:56.730297 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.730275    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/359cd30d-7981-4184-8a6f-f63a2f585517-tmp-dir\") pod \"dns-default-x2brk\" (UID: \"359cd30d-7981-4184-8a6f-f63a2f585517\") " pod="openshift-dns/dns-default-x2brk"
Sep 16 18:39:56.730445 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.730426    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/359cd30d-7981-4184-8a6f-f63a2f585517-config-volume\") pod \"dns-default-x2brk\" (UID: \"359cd30d-7981-4184-8a6f-f63a2f585517\") " pod="openshift-dns/dns-default-x2brk"
Sep 16 18:39:56.746459 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.746424    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9mdb\" (UniqueName: \"kubernetes.io/projected/359cd30d-7981-4184-8a6f-f63a2f585517-kube-api-access-h9mdb\") pod \"dns-default-x2brk\" (UID: \"359cd30d-7981-4184-8a6f-f63a2f585517\") " pod="openshift-dns/dns-default-x2brk"
Sep 16 18:39:56.830515 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.830472    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a00e71a-7efb-45c5-ac52-6e175ea6837a-cert\") pod \"ingress-canary-vb8t5\" (UID: \"1a00e71a-7efb-45c5-ac52-6e175ea6837a\") " pod="openshift-ingress-canary/ingress-canary-vb8t5"
Sep 16 18:39:56.830515 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.830513    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvlhx\" (UniqueName: \"kubernetes.io/projected/1a00e71a-7efb-45c5-ac52-6e175ea6837a-kube-api-access-kvlhx\") pod \"ingress-canary-vb8t5\" (UID: \"1a00e71a-7efb-45c5-ac52-6e175ea6837a\") " pod="openshift-ingress-canary/ingress-canary-vb8t5"
Sep 16 18:39:56.832990 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.832971    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a00e71a-7efb-45c5-ac52-6e175ea6837a-cert\") pod \"ingress-canary-vb8t5\" (UID: \"1a00e71a-7efb-45c5-ac52-6e175ea6837a\") " pod="openshift-ingress-canary/ingress-canary-vb8t5"
Sep 16 18:39:56.841165 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.841129    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvlhx\" (UniqueName: \"kubernetes.io/projected/1a00e71a-7efb-45c5-ac52-6e175ea6837a-kube-api-access-kvlhx\") pod \"ingress-canary-vb8t5\" (UID: \"1a00e71a-7efb-45c5-ac52-6e175ea6837a\") " pod="openshift-ingress-canary/ingress-canary-vb8t5"
Sep 16 18:39:56.896278 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:56.896238    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vb8t5"
Sep 16 18:39:57.234717 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:57.234680    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls\") pod \"dns-default-x2brk\" (UID: \"359cd30d-7981-4184-8a6f-f63a2f585517\") " pod="openshift-dns/dns-default-x2brk"
Sep 16 18:39:57.234884 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:57.234835    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:39:57.234931 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:57.234905    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls podName:359cd30d-7981-4184-8a6f-f63a2f585517 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:58.234888419 +0000 UTC m=+36.594406392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls") pod "dns-default-x2brk" (UID: "359cd30d-7981-4184-8a6f-f63a2f585517") : secret "dns-default-metrics-tls" not found
Sep 16 18:39:57.264756 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:57.264674    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vb8t5"]
Sep 16 18:39:57.287306 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:39:57.287255    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a00e71a_7efb_45c5_ac52_6e175ea6837a.slice/crio-afdce9f6d101aa8d5cd66e96278bee4a5b85972a88abdae018c29591bad54d7c WatchSource:0}: Error finding container afdce9f6d101aa8d5cd66e96278bee4a5b85972a88abdae018c29591bad54d7c: Status 404 returned error can't find the container with id afdce9f6d101aa8d5cd66e96278bee4a5b85972a88abdae018c29591bad54d7c
Sep 16 18:39:57.384844 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:57.384808    2565 generic.go:356] "Generic (PLEG): container finished" podID="f251ea42-1492-4bf4-942e-fdd522e5bbc5" containerID="a1b33f33f09672f13d54aa782e02615fe9d96e6b2b25a0bde5fd27ca84fc32d6" exitCode=0
Sep 16 18:39:57.385004 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:57.384888    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hhl4g" event={"ID":"f251ea42-1492-4bf4-942e-fdd522e5bbc5","Type":"ContainerDied","Data":"a1b33f33f09672f13d54aa782e02615fe9d96e6b2b25a0bde5fd27ca84fc32d6"}
Sep 16 18:39:57.386120 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:57.386093    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vb8t5" event={"ID":"1a00e71a-7efb-45c5-ac52-6e175ea6837a","Type":"ContainerStarted","Data":"afdce9f6d101aa8d5cd66e96278bee4a5b85972a88abdae018c29591bad54d7c"}
Sep 16 18:39:58.201417 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:58.201377    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:39:58.202184 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:58.201377    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:39:58.202184 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:58.201377    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:39:58.205634 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:58.205605    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:39:58.205786 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:58.205668    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vcswh\""
Sep 16 18:39:58.205786 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:58.205708    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:39:58.205786 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:58.205672    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:39:58.205786 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:58.205779    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5zf4c\""
Sep 16 18:39:58.205996 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:58.205856    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:39:58.241858 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:58.241826    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls\") pod \"dns-default-x2brk\" (UID: \"359cd30d-7981-4184-8a6f-f63a2f585517\") " pod="openshift-dns/dns-default-x2brk"
Sep 16 18:39:58.242036 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:58.242008    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:39:58.242175 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:39:58.242108    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls podName:359cd30d-7981-4184-8a6f-f63a2f585517 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:00.24208526 +0000 UTC m=+38.601603240 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls") pod "dns-default-x2brk" (UID: "359cd30d-7981-4184-8a6f-f63a2f585517") : secret "dns-default-metrics-tls" not found
Sep 16 18:39:58.392906 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:58.392861    2565 generic.go:356] "Generic (PLEG): container finished" podID="f251ea42-1492-4bf4-942e-fdd522e5bbc5" containerID="099852867dee51c177389cbe213446044d1fc2dab398419fecad1a563e15f68e" exitCode=0
Sep 16 18:39:58.393112 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:58.392925    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hhl4g" event={"ID":"f251ea42-1492-4bf4-942e-fdd522e5bbc5","Type":"ContainerDied","Data":"099852867dee51c177389cbe213446044d1fc2dab398419fecad1a563e15f68e"}
Sep 16 18:39:59.399308 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:59.399271    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hhl4g" event={"ID":"f251ea42-1492-4bf4-942e-fdd522e5bbc5","Type":"ContainerStarted","Data":"86b9322e051ecaf68d3d472e8c9143f4930fe112485b1d183caabc86a9cc757a"}
Sep 16 18:39:59.423612 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:39:59.423550    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hhl4g" podStartSLOduration=8.310933687 podStartE2EDuration="37.423532119s" totalImagesPullingTime="29.112598432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:59.422030992 +0000 UTC m=+37.781548998" watchObservedRunningTime="2026-09-16 18:39:59.423532119 +0000 UTC m=+37.783050117"
Sep 16 18:40:00.259402 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:00.259330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls\") pod \"dns-default-x2brk\" (UID: \"359cd30d-7981-4184-8a6f-f63a2f585517\") " pod="openshift-dns/dns-default-x2brk"
Sep 16 18:40:00.259582 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:00.259487    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:00.259582 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:00.259559    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls podName:359cd30d-7981-4184-8a6f-f63a2f585517 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:04.259540381 +0000 UTC m=+42.619058354 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls") pod "dns-default-x2brk" (UID: "359cd30d-7981-4184-8a6f-f63a2f585517") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:00.403004 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:00.402962    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vb8t5" event={"ID":"1a00e71a-7efb-45c5-ac52-6e175ea6837a","Type":"ContainerStarted","Data":"4a33ed739ac8b1a958d534f8cdbbcb1593696b56720a5a61016d8cbc12ecb587"}
Sep 16 18:40:00.421131 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:00.421017    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vb8t5" podStartSLOduration=2.033917066 podStartE2EDuration="4.420999507s" totalImagesPullingTime="2.387082441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:00.419929413 +0000 UTC m=+38.779447399" watchObservedRunningTime="2026-09-16 18:40:00.420999507 +0000 UTC m=+38.780517527"
Sep 16 18:40:04.287184 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:04.287141    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls\") pod \"dns-default-x2brk\" (UID: \"359cd30d-7981-4184-8a6f-f63a2f585517\") " pod="openshift-dns/dns-default-x2brk"
Sep 16 18:40:04.287608 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:04.287295    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:04.287608 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:04.287383    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls podName:359cd30d-7981-4184-8a6f-f63a2f585517 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:12.287363206 +0000 UTC m=+50.646881192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls") pod "dns-default-x2brk" (UID: "359cd30d-7981-4184-8a6f-f63a2f585517") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:12.343613 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:12.343562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls\") pod \"dns-default-x2brk\" (UID: \"359cd30d-7981-4184-8a6f-f63a2f585517\") " pod="openshift-dns/dns-default-x2brk"
Sep 16 18:40:12.344473 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:12.344445    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:12.344689 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:12.344675    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls podName:359cd30d-7981-4184-8a6f-f63a2f585517 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:28.344652384 +0000 UTC m=+66.704170378 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls") pod "dns-default-x2brk" (UID: "359cd30d-7981-4184-8a6f-f63a2f585517") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:22.380298 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:22.380271    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lpp26"
Sep 16 18:40:27.852536 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:27.852495    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs\") pod \"network-metrics-daemon-7r6xr\" (UID: \"2344014b-38cc-448f-9c9d-6b49d8e80b85\") " pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:40:27.855282 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:27.855249    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:40:27.863435 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:27.863400    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:40:27.863537 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:27.863499    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs podName:2344014b-38cc-448f-9c9d-6b49d8e80b85 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:31.863480964 +0000 UTC m=+130.222998941 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs") pod "network-metrics-daemon-7r6xr" (UID: "2344014b-38cc-448f-9c9d-6b49d8e80b85") : secret "metrics-daemon-secret" not found
Sep 16 18:40:27.953458 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:27.953417    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqfsl\" (UniqueName: \"kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl\") pod \"network-check-target-hqh7p\" (UID: \"dbb3e980-5ba7-4b12-82b2-1fa036ec00b7\") " pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:40:27.953561 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:27.953469    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret\") pod \"global-pull-secret-syncer-ddtnd\" (UID: \"cea41c21-51ad-46c1-8f7c-fc4dab1008e7\") " pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:40:27.957266 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:27.957239    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:40:27.957266 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:27.957239    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:40:27.966690 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:27.966667    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:40:27.967390 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:27.967371    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cea41c21-51ad-46c1-8f7c-fc4dab1008e7-original-pull-secret\") pod \"global-pull-secret-syncer-ddtnd\" (UID: \"cea41c21-51ad-46c1-8f7c-fc4dab1008e7\") " pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:40:27.976968 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:27.976942    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqfsl\" (UniqueName: \"kubernetes.io/projected/dbb3e980-5ba7-4b12-82b2-1fa036ec00b7-kube-api-access-xqfsl\") pod \"network-check-target-hqh7p\" (UID: \"dbb3e980-5ba7-4b12-82b2-1fa036ec00b7\") " pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:40:28.021768 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.021732    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7"]
Sep 16 18:40:28.025913 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.025893    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7"
Sep 16 18:40:28.030518 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.030491    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:28.030518 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.030507    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 18:40:28.030697 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.030633    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:28.031123 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.031100    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-v5vfz\""
Sep 16 18:40:28.044763 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.044727    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7"]
Sep 16 18:40:28.155457 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.155329    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn4wk\" (UniqueName: \"kubernetes.io/projected/9eab6685-d3cf-40d6-9616-340205432455-kube-api-access-rn4wk\") pod \"cluster-samples-operator-86499c89c4-tnjb7\" (UID: \"9eab6685-d3cf-40d6-9616-340205432455\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7"
Sep 16 18:40:28.155457 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.155402    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tnjb7\" (UID: \"9eab6685-d3cf-40d6-9616-340205432455\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7"
Sep 16 18:40:28.217444 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.217411    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5zf4c\""
Sep 16 18:40:28.224303 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.224271    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:40:28.226927 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.226902    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddtnd"
Sep 16 18:40:28.256090 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.256049    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn4wk\" (UniqueName: \"kubernetes.io/projected/9eab6685-d3cf-40d6-9616-340205432455-kube-api-access-rn4wk\") pod \"cluster-samples-operator-86499c89c4-tnjb7\" (UID: \"9eab6685-d3cf-40d6-9616-340205432455\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7"
Sep 16 18:40:28.256277 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.256111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tnjb7\" (UID: \"9eab6685-d3cf-40d6-9616-340205432455\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7"
Sep 16 18:40:28.256361 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:28.256288    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:40:28.256446 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:28.256432    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls podName:9eab6685-d3cf-40d6-9616-340205432455 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:28.756407633 +0000 UTC m=+67.115925626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tnjb7" (UID: "9eab6685-d3cf-40d6-9616-340205432455") : secret "samples-operator-tls" not found
Sep 16 18:40:28.272899 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.272864    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn4wk\" (UniqueName: \"kubernetes.io/projected/9eab6685-d3cf-40d6-9616-340205432455-kube-api-access-rn4wk\") pod \"cluster-samples-operator-86499c89c4-tnjb7\" (UID: \"9eab6685-d3cf-40d6-9616-340205432455\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7"
Sep 16 18:40:28.357252 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.357205    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls\") pod \"dns-default-x2brk\" (UID: \"359cd30d-7981-4184-8a6f-f63a2f585517\") " pod="openshift-dns/dns-default-x2brk"
Sep 16 18:40:28.357462 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:28.357332    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:28.357462 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:28.357410    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls podName:359cd30d-7981-4184-8a6f-f63a2f585517 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:00.357395467 +0000 UTC m=+98.716913453 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls") pod "dns-default-x2brk" (UID: "359cd30d-7981-4184-8a6f-f63a2f585517") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:28.529315 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.529276    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hqh7p"]
Sep 16 18:40:28.535841 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:40:28.535811    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb3e980_5ba7_4b12_82b2_1fa036ec00b7.slice/crio-8a96ce7c29deb887231fb0d51b8bb2e8cad62fc3c2ced747c19930325a593ba6 WatchSource:0}: Error finding container 8a96ce7c29deb887231fb0d51b8bb2e8cad62fc3c2ced747c19930325a593ba6: Status 404 returned error can't find the container with id 8a96ce7c29deb887231fb0d51b8bb2e8cad62fc3c2ced747c19930325a593ba6
Sep 16 18:40:28.545950 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.544923    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ddtnd"]
Sep 16 18:40:28.550572 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:40:28.550544    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea41c21_51ad_46c1_8f7c_fc4dab1008e7.slice/crio-b464ed7421071e36dee460281e008c7b52a55ce008aed4a228c518f0e7f32d1b WatchSource:0}: Error finding container b464ed7421071e36dee460281e008c7b52a55ce008aed4a228c518f0e7f32d1b: Status 404 returned error can't find the container with id b464ed7421071e36dee460281e008c7b52a55ce008aed4a228c518f0e7f32d1b
Sep 16 18:40:28.759698 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.759654    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tnjb7\" (UID: \"9eab6685-d3cf-40d6-9616-340205432455\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7"
Sep 16 18:40:28.759874 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:28.759823    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:40:28.759916 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:28.759894    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls podName:9eab6685-d3cf-40d6-9616-340205432455 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:29.759877802 +0000 UTC m=+68.119395775 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tnjb7" (UID: "9eab6685-d3cf-40d6-9616-340205432455") : secret "samples-operator-tls" not found
Sep 16 18:40:28.984433 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.984394    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"]
Sep 16 18:40:28.987637 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.987610    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:28.995421 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.995391    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:40:28.995580 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.995527    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:40:28.995580 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.995546    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:40:28.996744 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:28.996588    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7smv5\""
Sep 16 18:40:29.005143 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.005115    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:40:29.016760 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.016714    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"]
Sep 16 18:40:29.162801 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.162762    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-bound-sa-token\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.162995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.162811    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ab9989d-a434-4aa6-a97a-16410c87c1af-image-registry-private-configuration\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.162995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.162844    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-certificates\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.162995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.162897    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ab9989d-a434-4aa6-a97a-16410c87c1af-trusted-ca\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.162995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.162955    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g6pf\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-kube-api-access-8g6pf\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.163175 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.163014    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.163175 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.163084    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ab9989d-a434-4aa6-a97a-16410c87c1af-ca-trust-extracted\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.163175 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.163124    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ab9989d-a434-4aa6-a97a-16410c87c1af-installation-pull-secrets\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.264446 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.264063    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ab9989d-a434-4aa6-a97a-16410c87c1af-installation-pull-secrets\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.264446 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.264129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-bound-sa-token\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.264446 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.264162    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ab9989d-a434-4aa6-a97a-16410c87c1af-image-registry-private-configuration\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.264446 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.264194    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-certificates\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.264446 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.264218    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ab9989d-a434-4aa6-a97a-16410c87c1af-trusted-ca\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.264446 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.264380    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g6pf\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-kube-api-access-8g6pf\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.264446 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.264447    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.264915 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.264500    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ab9989d-a434-4aa6-a97a-16410c87c1af-ca-trust-extracted\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.264915 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:29.264840    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:29.264915 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:29.264864    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d74c7ddcb-wg6c6: secret "image-registry-tls" not found
Sep 16 18:40:29.265058 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.264934    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ab9989d-a434-4aa6-a97a-16410c87c1af-ca-trust-extracted\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.265058 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:29.264947    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls podName:9ab9989d-a434-4aa6-a97a-16410c87c1af nodeName:}" failed. No retries permitted until 2026-09-16 18:40:29.764922841 +0000 UTC m=+68.124440836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls") pod "image-registry-7d74c7ddcb-wg6c6" (UID: "9ab9989d-a434-4aa6-a97a-16410c87c1af") : secret "image-registry-tls" not found
Sep 16 18:40:29.265404 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.265334    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-certificates\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.265676 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.265647    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ab9989d-a434-4aa6-a97a-16410c87c1af-trusted-ca\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.267334 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.267310    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ab9989d-a434-4aa6-a97a-16410c87c1af-image-registry-private-configuration\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.268270 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.268243    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ab9989d-a434-4aa6-a97a-16410c87c1af-installation-pull-secrets\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.280393 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.280338    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g6pf\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-kube-api-access-8g6pf\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.280761 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.280738    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-bound-sa-token\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.476216 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.476163    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ddtnd" event={"ID":"cea41c21-51ad-46c1-8f7c-fc4dab1008e7","Type":"ContainerStarted","Data":"b464ed7421071e36dee460281e008c7b52a55ce008aed4a228c518f0e7f32d1b"}
Sep 16 18:40:29.477839 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.477796    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hqh7p" event={"ID":"dbb3e980-5ba7-4b12-82b2-1fa036ec00b7","Type":"ContainerStarted","Data":"8a96ce7c29deb887231fb0d51b8bb2e8cad62fc3c2ced747c19930325a593ba6"}
Sep 16 18:40:29.768257 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.768213    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tnjb7\" (UID: \"9eab6685-d3cf-40d6-9616-340205432455\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7"
Sep 16 18:40:29.768484 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:29.768296    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:29.768484 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:29.768391    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:40:29.768484 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:29.768469    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls podName:9eab6685-d3cf-40d6-9616-340205432455 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:31.768450833 +0000 UTC m=+70.127968827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tnjb7" (UID: "9eab6685-d3cf-40d6-9616-340205432455") : secret "samples-operator-tls" not found
Sep 16 18:40:29.768484 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:29.768473    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:29.768703 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:29.768499    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d74c7ddcb-wg6c6: secret "image-registry-tls" not found
Sep 16 18:40:29.768703 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:29.768562    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls podName:9ab9989d-a434-4aa6-a97a-16410c87c1af nodeName:}" failed. No retries permitted until 2026-09-16 18:40:30.768542021 +0000 UTC m=+69.128060008 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls") pod "image-registry-7d74c7ddcb-wg6c6" (UID: "9ab9989d-a434-4aa6-a97a-16410c87c1af") : secret "image-registry-tls" not found
Sep 16 18:40:30.776791 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:30.776683    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:30.777202 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:30.776841    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:30.777202 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:30.776860    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d74c7ddcb-wg6c6: secret "image-registry-tls" not found
Sep 16 18:40:30.777202 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:30.776926    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls podName:9ab9989d-a434-4aa6-a97a-16410c87c1af nodeName:}" failed. No retries permitted until 2026-09-16 18:40:32.776906107 +0000 UTC m=+71.136424094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls") pod "image-registry-7d74c7ddcb-wg6c6" (UID: "9ab9989d-a434-4aa6-a97a-16410c87c1af") : secret "image-registry-tls" not found
Sep 16 18:40:31.783747 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:31.783698    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tnjb7\" (UID: \"9eab6685-d3cf-40d6-9616-340205432455\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7"
Sep 16 18:40:31.784155 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:31.783845    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:40:31.784155 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:31.783922    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls podName:9eab6685-d3cf-40d6-9616-340205432455 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:35.783897624 +0000 UTC m=+74.143415620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tnjb7" (UID: "9eab6685-d3cf-40d6-9616-340205432455") : secret "samples-operator-tls" not found
Sep 16 18:40:32.569323 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.569282    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-zvt9g"]
Sep 16 18:40:32.572460 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.572427    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-zvt9g"
Sep 16 18:40:32.575691 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.575440    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7cgs2\""
Sep 16 18:40:32.582937 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.582904    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-zvt9g"]
Sep 16 18:40:32.588770 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.588725    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kgcj\" (UniqueName: \"kubernetes.io/projected/c7b187aa-f9c8-423c-b0aa-3e63ab8187bd-kube-api-access-2kgcj\") pod \"network-check-source-6f8b768455-zvt9g\" (UID: \"c7b187aa-f9c8-423c-b0aa-3e63ab8187bd\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-zvt9g"
Sep 16 18:40:32.599319 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.599284    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ww4bs"]
Sep 16 18:40:32.602631 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.602606    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:32.605132 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.605109    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:40:32.605495 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.605475    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:40:32.605495 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.605488    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-txfhn\""
Sep 16 18:40:32.605627 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.605529    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:40:32.607532 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.607507    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:40:32.621211 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.621147    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ww4bs"]
Sep 16 18:40:32.690091 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.690041    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d1a62b24-aeeb-43c1-8279-3d4e479eb481-crio-socket\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:32.690286 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.690147    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1a62b24-aeeb-43c1-8279-3d4e479eb481-data-volume\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:32.690286 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.690176    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:32.690286 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.690200    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khbbf\" (UniqueName: \"kubernetes.io/projected/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-api-access-khbbf\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:32.690286 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.690223    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kgcj\" (UniqueName: \"kubernetes.io/projected/c7b187aa-f9c8-423c-b0aa-3e63ab8187bd-kube-api-access-2kgcj\") pod \"network-check-source-6f8b768455-zvt9g\" (UID: \"c7b187aa-f9c8-423c-b0aa-3e63ab8187bd\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-zvt9g"
Sep 16 18:40:32.690496 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.690340    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:32.715025 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.714992    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kgcj\" (UniqueName: \"kubernetes.io/projected/c7b187aa-f9c8-423c-b0aa-3e63ab8187bd-kube-api-access-2kgcj\") pod \"network-check-source-6f8b768455-zvt9g\" (UID: \"c7b187aa-f9c8-423c-b0aa-3e63ab8187bd\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-zvt9g"
Sep 16 18:40:32.790987 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.790947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:32.791508 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.791010    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1a62b24-aeeb-43c1-8279-3d4e479eb481-data-volume\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:32.791508 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.791041    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:32.791508 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.791070    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khbbf\" (UniqueName: \"kubernetes.io/projected/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-api-access-khbbf\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:32.791508 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:32.791112    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:32.791508 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:32.791128    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d74c7ddcb-wg6c6: secret "image-registry-tls" not found
Sep 16 18:40:32.791508 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.791135    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:32.791508 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.791163    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d1a62b24-aeeb-43c1-8279-3d4e479eb481-crio-socket\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:32.791508 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:32.791192    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls podName:9ab9989d-a434-4aa6-a97a-16410c87c1af nodeName:}" failed. No retries permitted until 2026-09-16 18:40:36.791171904 +0000 UTC m=+75.150689894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls") pod "image-registry-7d74c7ddcb-wg6c6" (UID: "9ab9989d-a434-4aa6-a97a-16410c87c1af") : secret "image-registry-tls" not found
Sep 16 18:40:32.791508 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:32.791218    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:32.791508 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:32.791272    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:40:32.791508 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:32.791302    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls podName:d1a62b24-aeeb-43c1-8279-3d4e479eb481 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:33.291283263 +0000 UTC m=+71.650801489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ww4bs" (UID: "d1a62b24-aeeb-43c1-8279-3d4e479eb481") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:32.791508 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.791227    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d1a62b24-aeeb-43c1-8279-3d4e479eb481-crio-socket\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:32.791508 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:32.791326    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-rbac-proxy-cm podName:d1a62b24-aeeb-43c1-8279-3d4e479eb481 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:33.291315053 +0000 UTC m=+71.650833025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ww4bs" (UID: "d1a62b24-aeeb-43c1-8279-3d4e479eb481") : configmap "kube-rbac-proxy" not found
Sep 16 18:40:32.791508 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.791485    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1a62b24-aeeb-43c1-8279-3d4e479eb481-data-volume\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:32.801804 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.801777    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khbbf\" (UniqueName: \"kubernetes.io/projected/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-api-access-khbbf\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:32.884325 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:32.884246    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-zvt9g"
Sep 16 18:40:33.200181 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:33.200057    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-zvt9g"]
Sep 16 18:40:33.202704 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:40:33.202675    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b187aa_f9c8_423c_b0aa_3e63ab8187bd.slice/crio-59d0475d7bd3941da4580efb442c1e3a55af8aaaec79dda3df802a9223374235 WatchSource:0}: Error finding container 59d0475d7bd3941da4580efb442c1e3a55af8aaaec79dda3df802a9223374235: Status 404 returned error can't find the container with id 59d0475d7bd3941da4580efb442c1e3a55af8aaaec79dda3df802a9223374235
Sep 16 18:40:33.295686 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:33.295641    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:33.295936 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:33.295797    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:33.295936 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:33.295843    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:33.295936 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:33.295869    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls podName:d1a62b24-aeeb-43c1-8279-3d4e479eb481 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:34.295849299 +0000 UTC m=+72.655367290 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ww4bs" (UID: "d1a62b24-aeeb-43c1-8279-3d4e479eb481") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:33.295936 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:33.295924    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:40:33.296153 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:33.295983    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-rbac-proxy-cm podName:d1a62b24-aeeb-43c1-8279-3d4e479eb481 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:34.295966549 +0000 UTC m=+72.655484523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ww4bs" (UID: "d1a62b24-aeeb-43c1-8279-3d4e479eb481") : configmap "kube-rbac-proxy" not found
Sep 16 18:40:33.489589 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:33.489551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ddtnd" event={"ID":"cea41c21-51ad-46c1-8f7c-fc4dab1008e7","Type":"ContainerStarted","Data":"bc6f9c1a890198304bdfcfcb34bf12dbb3083ed9c40477431728f062f89a9501"}
Sep 16 18:40:33.491193 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:33.491156    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hqh7p" event={"ID":"dbb3e980-5ba7-4b12-82b2-1fa036ec00b7","Type":"ContainerStarted","Data":"a98b9c17a4ecabae26b63b0af23dad593ced22256051035099f70e0fb4b9a056"}
Sep 16 18:40:33.491369 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:33.491217    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:40:33.492715 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:33.492689    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-zvt9g" event={"ID":"c7b187aa-f9c8-423c-b0aa-3e63ab8187bd","Type":"ContainerStarted","Data":"5436c44a608179553d17a6f8a5b4a6285ca34afdcd32b439b55423b1d803bdbc"}
Sep 16 18:40:33.492715 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:33.492743    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-zvt9g" event={"ID":"c7b187aa-f9c8-423c-b0aa-3e63ab8187bd","Type":"ContainerStarted","Data":"59d0475d7bd3941da4580efb442c1e3a55af8aaaec79dda3df802a9223374235"}
Sep 16 18:40:33.511475 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:33.511418    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ddtnd" podStartSLOduration=67.126639237 podStartE2EDuration="1m11.511398424s" totalImagesPullingTime="4.384759187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:33.510888335 +0000 UTC m=+71.870406330" watchObservedRunningTime="2026-09-16 18:40:33.511398424 +0000 UTC m=+71.870916420"
Sep 16 18:40:33.532118 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:33.532062    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hqh7p" podStartSLOduration=67.138375992 podStartE2EDuration="1m11.532045236s" totalImagesPullingTime="4.393669244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:33.531781631 +0000 UTC m=+71.891299625" watchObservedRunningTime="2026-09-16 18:40:33.532045236 +0000 UTC m=+71.891563222"
Sep 16 18:40:33.554995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:33.554938    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-zvt9g" podStartSLOduration=1.554919116 podStartE2EDuration="1.554919116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:33.5530362 +0000 UTC m=+71.912554194" watchObservedRunningTime="2026-09-16 18:40:33.554919116 +0000 UTC m=+71.914437111"
Sep 16 18:40:34.301677 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:34.301639    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:34.302083 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:34.301727    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:34.302083 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:34.301785    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:40:34.302083 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:34.301825    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:34.302083 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:34.301875    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-rbac-proxy-cm podName:d1a62b24-aeeb-43c1-8279-3d4e479eb481 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:36.301852977 +0000 UTC m=+74.661370966 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ww4bs" (UID: "d1a62b24-aeeb-43c1-8279-3d4e479eb481") : configmap "kube-rbac-proxy" not found
Sep 16 18:40:34.302083 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:34.301895    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls podName:d1a62b24-aeeb-43c1-8279-3d4e479eb481 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:36.301885863 +0000 UTC m=+74.661403836 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ww4bs" (UID: "d1a62b24-aeeb-43c1-8279-3d4e479eb481") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:34.691189 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:34.691112    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tch8d_fe676eb4-48c2-485b-9752-2198abad740a/dns-node-resolver/0.log"
Sep 16 18:40:35.291596 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:35.291568    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8pj92_d1cdba36-53e5-400a-9fdd-7c62a3fe0086/node-ca/0.log"
Sep 16 18:40:35.809523 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:35.809474    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tnjb7\" (UID: \"9eab6685-d3cf-40d6-9616-340205432455\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7"
Sep 16 18:40:35.809901 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:35.809621    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:40:35.809901 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:35.809690    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls podName:9eab6685-d3cf-40d6-9616-340205432455 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:43.809673915 +0000 UTC m=+82.169191888 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tnjb7" (UID: "9eab6685-d3cf-40d6-9616-340205432455") : secret "samples-operator-tls" not found
Sep 16 18:40:36.313145 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:36.313101    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:36.313331 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:36.313170    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:36.313331 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:36.313241    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:36.313331 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:36.313311    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls podName:d1a62b24-aeeb-43c1-8279-3d4e479eb481 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:40.313293048 +0000 UTC m=+78.672811021 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ww4bs" (UID: "d1a62b24-aeeb-43c1-8279-3d4e479eb481") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:36.313331 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:36.313243    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:40:36.313501 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:36.313396    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-rbac-proxy-cm podName:d1a62b24-aeeb-43c1-8279-3d4e479eb481 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:40.313384352 +0000 UTC m=+78.672902325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ww4bs" (UID: "d1a62b24-aeeb-43c1-8279-3d4e479eb481") : configmap "kube-rbac-proxy" not found
Sep 16 18:40:36.492018 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:36.491986    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vb8t5_1a00e71a-7efb-45c5-ac52-6e175ea6837a/serve-healthcheck-canary/0.log"
Sep 16 18:40:36.815782 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:36.815740    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:36.816165 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:36.815886    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:36.816165 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:36.815908    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d74c7ddcb-wg6c6: secret "image-registry-tls" not found
Sep 16 18:40:36.816165 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:36.815969    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls podName:9ab9989d-a434-4aa6-a97a-16410c87c1af nodeName:}" failed. No retries permitted until 2026-09-16 18:40:44.815952844 +0000 UTC m=+83.175470817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls") pod "image-registry-7d74c7ddcb-wg6c6" (UID: "9ab9989d-a434-4aa6-a97a-16410c87c1af") : secret "image-registry-tls" not found
Sep 16 18:40:37.625386 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:37.625336    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-9tgwr"]
Sep 16 18:40:37.629501 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:37.629470    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9tgwr"
Sep 16 18:40:37.632192 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:37.632166    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 18:40:37.632315 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:37.632179    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:37.633191 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:37.633176    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bjxl2\""
Sep 16 18:40:37.639778 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:37.639751    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-9tgwr"]
Sep 16 18:40:37.723588 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:37.723547    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkjkf\" (UniqueName: \"kubernetes.io/projected/a266096e-4a22-411c-a43f-2d68a3dd93ed-kube-api-access-gkjkf\") pod \"migrator-8b4d697c5-9tgwr\" (UID: \"a266096e-4a22-411c-a43f-2d68a3dd93ed\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9tgwr"
Sep 16 18:40:37.824214 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:37.824171    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkjkf\" (UniqueName: \"kubernetes.io/projected/a266096e-4a22-411c-a43f-2d68a3dd93ed-kube-api-access-gkjkf\") pod \"migrator-8b4d697c5-9tgwr\" (UID: \"a266096e-4a22-411c-a43f-2d68a3dd93ed\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9tgwr"
Sep 16 18:40:37.834472 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:37.834436    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkjkf\" (UniqueName: \"kubernetes.io/projected/a266096e-4a22-411c-a43f-2d68a3dd93ed-kube-api-access-gkjkf\") pod \"migrator-8b4d697c5-9tgwr\" (UID: \"a266096e-4a22-411c-a43f-2d68a3dd93ed\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9tgwr"
Sep 16 18:40:37.938894 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:37.938803    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9tgwr"
Sep 16 18:40:38.210088 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:38.210002    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-9tgwr"]
Sep 16 18:40:38.213401 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:40:38.213373    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda266096e_4a22_411c_a43f_2d68a3dd93ed.slice/crio-06404858605fb03808845c394e328c83f004fad26770d91ee88d8ce2965a39f5 WatchSource:0}: Error finding container 06404858605fb03808845c394e328c83f004fad26770d91ee88d8ce2965a39f5: Status 404 returned error can't find the container with id 06404858605fb03808845c394e328c83f004fad26770d91ee88d8ce2965a39f5
Sep 16 18:40:38.509054 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:38.509020    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9tgwr" event={"ID":"a266096e-4a22-411c-a43f-2d68a3dd93ed","Type":"ContainerStarted","Data":"06404858605fb03808845c394e328c83f004fad26770d91ee88d8ce2965a39f5"}
Sep 16 18:40:40.342829 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:40.342790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:40.343181 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:40.342862    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:40.343181 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:40.342945    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:40:40.343181 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:40.342955    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:40.343181 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:40.343015    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-rbac-proxy-cm podName:d1a62b24-aeeb-43c1-8279-3d4e479eb481 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:48.342999176 +0000 UTC m=+86.702517165 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ww4bs" (UID: "d1a62b24-aeeb-43c1-8279-3d4e479eb481") : configmap "kube-rbac-proxy" not found
Sep 16 18:40:40.343181 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:40.343030    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls podName:d1a62b24-aeeb-43c1-8279-3d4e479eb481 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:48.34302454 +0000 UTC m=+86.702542513 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ww4bs" (UID: "d1a62b24-aeeb-43c1-8279-3d4e479eb481") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:40.517564 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:40.517526    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9tgwr" event={"ID":"a266096e-4a22-411c-a43f-2d68a3dd93ed","Type":"ContainerStarted","Data":"83387bdd27f7d33d4411034aaddcaee3086852f67f2687c2bb4a39270fb29602"}
Sep 16 18:40:40.517714 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:40.517582    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9tgwr" event={"ID":"a266096e-4a22-411c-a43f-2d68a3dd93ed","Type":"ContainerStarted","Data":"50bb92ca675613fe24c5aa7ee2ad39ee9888a9bf78bf30a2567092d451cf895e"}
Sep 16 18:40:40.550617 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:40.550555    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-9tgwr" podStartSLOduration=2.001934137 podStartE2EDuration="3.55054041s" totalImagesPullingTime="1.548606273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:40.550365187 +0000 UTC m=+78.909883181" watchObservedRunningTime="2026-09-16 18:40:40.55054041 +0000 UTC m=+78.910058404"
Sep 16 18:40:40.836143 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:40.836105    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l"]
Sep 16 18:40:40.839137 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:40.839119    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l"
Sep 16 18:40:40.845458 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:40.845065    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:40:40.845458 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:40.845118    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:40:40.845458 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:40.845160    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-tnxv2\""
Sep 16 18:40:40.846638 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:40.846620    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 18:40:40.847254 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:40.847234    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88d4b2eb-6870-45ee-848e-1a64e8d8b37e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l\" (UID: \"88d4b2eb-6870-45ee-848e-1a64e8d8b37e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l"
Sep 16 18:40:40.847298 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:40.847276    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdqnm\" (UniqueName: \"kubernetes.io/projected/88d4b2eb-6870-45ee-848e-1a64e8d8b37e-kube-api-access-cdqnm\") pod \"managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l\" (UID: \"88d4b2eb-6870-45ee-848e-1a64e8d8b37e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l"
Sep 16 18:40:40.849788 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:40.849773    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:40:40.858899 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:40.858869    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l"]
Sep 16 18:40:40.947807 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:40.947763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdqnm\" (UniqueName: \"kubernetes.io/projected/88d4b2eb-6870-45ee-848e-1a64e8d8b37e-kube-api-access-cdqnm\") pod \"managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l\" (UID: \"88d4b2eb-6870-45ee-848e-1a64e8d8b37e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l"
Sep 16 18:40:40.947964 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:40.947898    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88d4b2eb-6870-45ee-848e-1a64e8d8b37e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l\" (UID: \"88d4b2eb-6870-45ee-848e-1a64e8d8b37e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l"
Sep 16 18:40:40.950420 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:40.950387    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88d4b2eb-6870-45ee-848e-1a64e8d8b37e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l\" (UID: \"88d4b2eb-6870-45ee-848e-1a64e8d8b37e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l"
Sep 16 18:40:40.958110 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:40.958080    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdqnm\" (UniqueName: \"kubernetes.io/projected/88d4b2eb-6870-45ee-848e-1a64e8d8b37e-kube-api-access-cdqnm\") pod \"managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l\" (UID: \"88d4b2eb-6870-45ee-848e-1a64e8d8b37e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l"
Sep 16 18:40:41.171362 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:41.171252    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l"
Sep 16 18:40:41.458210 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:41.458172    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l"]
Sep 16 18:40:41.458566 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:40:41.458219    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88d4b2eb_6870_45ee_848e_1a64e8d8b37e.slice/crio-bd3872934f015586a2d0766f93c2783a8361d2aaa4e14a4e8edbe2433268dba4 WatchSource:0}: Error finding container bd3872934f015586a2d0766f93c2783a8361d2aaa4e14a4e8edbe2433268dba4: Status 404 returned error can't find the container with id bd3872934f015586a2d0766f93c2783a8361d2aaa4e14a4e8edbe2433268dba4
Sep 16 18:40:41.522910 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:41.522871    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l" event={"ID":"88d4b2eb-6870-45ee-848e-1a64e8d8b37e","Type":"ContainerStarted","Data":"bd3872934f015586a2d0766f93c2783a8361d2aaa4e14a4e8edbe2433268dba4"}
Sep 16 18:40:43.873425 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:43.873366    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tnjb7\" (UID: \"9eab6685-d3cf-40d6-9616-340205432455\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7"
Sep 16 18:40:43.873800 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:43.873461    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:40:43.873800 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:43.873540    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls podName:9eab6685-d3cf-40d6-9616-340205432455 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:59.873521915 +0000 UTC m=+98.233039909 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-tnjb7" (UID: "9eab6685-d3cf-40d6-9616-340205432455") : secret "samples-operator-tls" not found
Sep 16 18:40:44.535507 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:44.535376    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l" event={"ID":"88d4b2eb-6870-45ee-848e-1a64e8d8b37e","Type":"ContainerStarted","Data":"78a0874095dca007854580b7cff328d7a7ebd169dfcbfed99b2cb2cabd1d45ba"}
Sep 16 18:40:44.566280 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:44.566203    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cfd4b94dd-bkj4l" podStartSLOduration=2.344884075 podStartE2EDuration="4.566186208s" totalImagesPullingTime="2.221302133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:44.56228673 +0000 UTC m=+82.921804726" watchObservedRunningTime="2026-09-16 18:40:44.566186208 +0000 UTC m=+82.925704203"
Sep 16 18:40:44.880704 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:44.880587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:40:44.881107 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:44.880753    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:44.881107 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:44.880786    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d74c7ddcb-wg6c6: secret "image-registry-tls" not found
Sep 16 18:40:44.881107 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:44.880848    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls podName:9ab9989d-a434-4aa6-a97a-16410c87c1af nodeName:}" failed. No retries permitted until 2026-09-16 18:41:00.880831473 +0000 UTC m=+99.240349468 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls") pod "image-registry-7d74c7ddcb-wg6c6" (UID: "9ab9989d-a434-4aa6-a97a-16410c87c1af") : secret "image-registry-tls" not found
Sep 16 18:40:48.409008 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:48.408975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:48.409447 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:48.409050    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:48.409447 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:48.409151    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:48.409447 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:40:48.409236    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls podName:d1a62b24-aeeb-43c1-8279-3d4e479eb481 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:04.409212249 +0000 UTC m=+102.768730239 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ww4bs" (UID: "d1a62b24-aeeb-43c1-8279-3d4e479eb481") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:40:48.409667 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:48.409643    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1a62b24-aeeb-43c1-8279-3d4e479eb481-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:40:59.897804 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:59.897740    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tnjb7\" (UID: \"9eab6685-d3cf-40d6-9616-340205432455\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7"
Sep 16 18:40:59.900234 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:40:59.900211    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9eab6685-d3cf-40d6-9616-340205432455-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-tnjb7\" (UID: \"9eab6685-d3cf-40d6-9616-340205432455\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7"
Sep 16 18:41:00.135123 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:00.135083    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7"
Sep 16 18:41:00.401262 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:00.401221    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls\") pod \"dns-default-x2brk\" (UID: \"359cd30d-7981-4184-8a6f-f63a2f585517\") " pod="openshift-dns/dns-default-x2brk"
Sep 16 18:41:00.403573 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:00.403551    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/359cd30d-7981-4184-8a6f-f63a2f585517-metrics-tls\") pod \"dns-default-x2brk\" (UID: \"359cd30d-7981-4184-8a6f-f63a2f585517\") " pod="openshift-dns/dns-default-x2brk"
Sep 16 18:41:00.406286 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:00.406245    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7"]
Sep 16 18:41:00.478540 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:00.478507    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vxh4z\""
Sep 16 18:41:00.486025 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:00.486000    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x2brk"
Sep 16 18:41:00.586624 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:00.586588    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7" event={"ID":"9eab6685-d3cf-40d6-9616-340205432455","Type":"ContainerStarted","Data":"713661dc3bca440617e79e57d5d077401d574d3b608045892aedc9c021b5a9b9"}
Sep 16 18:41:00.768593 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:00.768560    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x2brk"]
Sep 16 18:41:00.771874 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:41:00.771843    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359cd30d_7981_4184_8a6f_f63a2f585517.slice/crio-e7e29e63d2a487988486566e027e64706988961eeac62a70d76f2abe20c53f46 WatchSource:0}: Error finding container e7e29e63d2a487988486566e027e64706988961eeac62a70d76f2abe20c53f46: Status 404 returned error can't find the container with id e7e29e63d2a487988486566e027e64706988961eeac62a70d76f2abe20c53f46
Sep 16 18:41:00.904182 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:00.904144    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:41:00.906791 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:00.906763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls\") pod \"image-registry-7d74c7ddcb-wg6c6\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") " pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:41:01.099945 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:01.099840    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:41:01.377266 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:01.377170    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"]
Sep 16 18:41:01.382177 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:41:01.382141    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab9989d_a434_4aa6_a97a_16410c87c1af.slice/crio-cca7e9df8173f15033164c9fa08ebca4d43893e4b094b557f9146ffca7f9da54 WatchSource:0}: Error finding container cca7e9df8173f15033164c9fa08ebca4d43893e4b094b557f9146ffca7f9da54: Status 404 returned error can't find the container with id cca7e9df8173f15033164c9fa08ebca4d43893e4b094b557f9146ffca7f9da54
Sep 16 18:41:01.592235 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:01.592189    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x2brk" event={"ID":"359cd30d-7981-4184-8a6f-f63a2f585517","Type":"ContainerStarted","Data":"e7e29e63d2a487988486566e027e64706988961eeac62a70d76f2abe20c53f46"}
Sep 16 18:41:01.594459 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:01.594426    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6" event={"ID":"9ab9989d-a434-4aa6-a97a-16410c87c1af","Type":"ContainerStarted","Data":"3efd8ed006bd2a95a5db00483002efc298d33d1a4b6a38681926a6f2da7e6f5a"}
Sep 16 18:41:01.594618 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:01.594475    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6" event={"ID":"9ab9989d-a434-4aa6-a97a-16410c87c1af","Type":"ContainerStarted","Data":"cca7e9df8173f15033164c9fa08ebca4d43893e4b094b557f9146ffca7f9da54"}
Sep 16 18:41:01.594883 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:01.594832    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:41:01.620820 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:01.620681    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6" podStartSLOduration=33.620658106 podStartE2EDuration="33.620658106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:01.619726233 +0000 UTC m=+99.979244229" watchObservedRunningTime="2026-09-16 18:41:01.620658106 +0000 UTC m=+99.980176102"
Sep 16 18:41:03.603197 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:03.603151    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7" event={"ID":"9eab6685-d3cf-40d6-9616-340205432455","Type":"ContainerStarted","Data":"ee66a3345223f9197e291e6b0b9efec7fc89595b0677c96b989c242ab6e6f73f"}
Sep 16 18:41:03.603710 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:03.603221    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7" event={"ID":"9eab6685-d3cf-40d6-9616-340205432455","Type":"ContainerStarted","Data":"0306cf997d11170b6f70d28a556efa81c5bb8b723604c698c7574daa59d6785d"}
Sep 16 18:41:03.604976 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:03.604945    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x2brk" event={"ID":"359cd30d-7981-4184-8a6f-f63a2f585517","Type":"ContainerStarted","Data":"6cd0664d66fab7b00fff68c22528b1038b3f9cc865183db94cf04b775b3e38a2"}
Sep 16 18:41:03.605082 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:03.604999    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x2brk" event={"ID":"359cd30d-7981-4184-8a6f-f63a2f585517","Type":"ContainerStarted","Data":"547c97a652a0e52c73d034f50e62973f1f0e61df7eb9c17c75992516217f83c1"}
Sep 16 18:41:03.605082 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:03.605065    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x2brk"
Sep 16 18:41:03.646243 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:03.646189    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-x2brk" podStartSLOduration=65.624584614 podStartE2EDuration="1m7.646171076s" totalImagesPullingTime="2.021586462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:03.646011221 +0000 UTC m=+102.005529228" watchObservedRunningTime="2026-09-16 18:41:03.646171076 +0000 UTC m=+102.005689072"
Sep 16 18:41:03.646958 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:03.646926    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-tnjb7" podStartSLOduration=34.302129808 podStartE2EDuration="36.646917012s" totalImagesPullingTime="2.344787204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:03.624852481 +0000 UTC m=+101.984370475" watchObservedRunningTime="2026-09-16 18:41:03.646917012 +0000 UTC m=+102.006435007"
Sep 16 18:41:04.434832 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:04.434787    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:41:04.437296 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:04.437262    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1a62b24-aeeb-43c1-8279-3d4e479eb481-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ww4bs\" (UID: \"d1a62b24-aeeb-43c1-8279-3d4e479eb481\") " pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:41:04.500358 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:04.500314    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hqh7p"
Sep 16 18:41:04.713096 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:04.713005    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ww4bs"
Sep 16 18:41:05.018062 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.018029    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ww4bs"]
Sep 16 18:41:05.019414 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:41:05.019388    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a62b24_aeeb_43c1_8279_3d4e479eb481.slice/crio-f6383f94b32cf7d69588e8fa4f3c88afd9f9803f6ec946546041cd78db49a7cf WatchSource:0}: Error finding container f6383f94b32cf7d69588e8fa4f3c88afd9f9803f6ec946546041cd78db49a7cf: Status 404 returned error can't find the container with id f6383f94b32cf7d69588e8fa4f3c88afd9f9803f6ec946546041cd78db49a7cf
Sep 16 18:41:05.569805 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.569770    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-ts6zv"]
Sep 16 18:41:05.582915 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.582885    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ts6zv"
Sep 16 18:41:05.601162 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.599923    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-ts6zv"]
Sep 16 18:41:05.611515 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.611484    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 18:41:05.612439 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.612415    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-bbhsj\""
Sep 16 18:41:05.612583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.612445    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 18:41:05.613156 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.613129    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ww4bs" event={"ID":"d1a62b24-aeeb-43c1-8279-3d4e479eb481","Type":"ContainerStarted","Data":"b6a432ec1cd5deaf4b58e0e5c5cf688e3597eb144547a523170a7642c2814a26"}
Sep 16 18:41:05.613243 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.613185    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ww4bs" event={"ID":"d1a62b24-aeeb-43c1-8279-3d4e479eb481","Type":"ContainerStarted","Data":"f6383f94b32cf7d69588e8fa4f3c88afd9f9803f6ec946546041cd78db49a7cf"}
Sep 16 18:41:05.643433 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.643396    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5acc4528-30cd-4e26-a485-02ba031ce861-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-ts6zv\" (UID: \"5acc4528-30cd-4e26-a485-02ba031ce861\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ts6zv"
Sep 16 18:41:05.643555 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.643523    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5acc4528-30cd-4e26-a485-02ba031ce861-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-ts6zv\" (UID: \"5acc4528-30cd-4e26-a485-02ba031ce861\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ts6zv"
Sep 16 18:41:05.724965 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.724929    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"]
Sep 16 18:41:05.744139 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.744105    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5acc4528-30cd-4e26-a485-02ba031ce861-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-ts6zv\" (UID: \"5acc4528-30cd-4e26-a485-02ba031ce861\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ts6zv"
Sep 16 18:41:05.744289 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.744155    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5acc4528-30cd-4e26-a485-02ba031ce861-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-ts6zv\" (UID: \"5acc4528-30cd-4e26-a485-02ba031ce861\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ts6zv"
Sep 16 18:41:05.744786 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.744765    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5acc4528-30cd-4e26-a485-02ba031ce861-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-ts6zv\" (UID: \"5acc4528-30cd-4e26-a485-02ba031ce861\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ts6zv"
Sep 16 18:41:05.746678 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.746653    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5acc4528-30cd-4e26-a485-02ba031ce861-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-ts6zv\" (UID: \"5acc4528-30cd-4e26-a485-02ba031ce861\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ts6zv"
Sep 16 18:41:05.768101 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.768063    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-56dzj"]
Sep 16 18:41:05.782356 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.782315    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-56dzj"
Sep 16 18:41:05.804246 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.804208    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-56dzj"]
Sep 16 18:41:05.810700 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.810669    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-j99sw\""
Sep 16 18:41:05.817096 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.817065    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 18:41:05.845244 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.845155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d84d2a1b-19d2-41cf-8ff2-510c1e37705c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-56dzj\" (UID: \"d84d2a1b-19d2-41cf-8ff2-510c1e37705c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-56dzj"
Sep 16 18:41:05.891807 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.891772    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ts6zv"
Sep 16 18:41:05.946239 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.946205    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d84d2a1b-19d2-41cf-8ff2-510c1e37705c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-56dzj\" (UID: \"d84d2a1b-19d2-41cf-8ff2-510c1e37705c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-56dzj"
Sep 16 18:41:05.948792 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:05.948765    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d84d2a1b-19d2-41cf-8ff2-510c1e37705c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-56dzj\" (UID: \"d84d2a1b-19d2-41cf-8ff2-510c1e37705c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-56dzj"
Sep 16 18:41:06.091816 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.091779    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-56dzj"
Sep 16 18:41:06.106648 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.106625    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68f546c58-hqpr7"]
Sep 16 18:41:06.120112 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.120084    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.135200 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.135168    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68f546c58-hqpr7"]
Sep 16 18:41:06.195168 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.195129    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-ts6zv"]
Sep 16 18:41:06.198157 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:41:06.198125    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5acc4528_30cd_4e26_a485_02ba031ce861.slice/crio-b37603555f0d26d574a83590d9724794d620bc54d9b6f985c7e7ffea2da07845 WatchSource:0}: Error finding container b37603555f0d26d574a83590d9724794d620bc54d9b6f985c7e7ffea2da07845: Status 404 returned error can't find the container with id b37603555f0d26d574a83590d9724794d620bc54d9b6f985c7e7ffea2da07845
Sep 16 18:41:06.248901 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.248872    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjsf2\" (UniqueName: \"kubernetes.io/projected/4354a2be-2116-4f59-b84c-93060dd44d08-kube-api-access-wjsf2\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.248982 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.248947    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4354a2be-2116-4f59-b84c-93060dd44d08-registry-tls\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.249045 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.248982    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4354a2be-2116-4f59-b84c-93060dd44d08-bound-sa-token\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.249183 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.249160    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4354a2be-2116-4f59-b84c-93060dd44d08-installation-pull-secrets\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.249242 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.249207    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4354a2be-2116-4f59-b84c-93060dd44d08-image-registry-private-configuration\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.249277 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.249239    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4354a2be-2116-4f59-b84c-93060dd44d08-ca-trust-extracted\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.249277 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.249265    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4354a2be-2116-4f59-b84c-93060dd44d08-trusted-ca\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.249373 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.249293    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4354a2be-2116-4f59-b84c-93060dd44d08-registry-certificates\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.350007 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.349970    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4354a2be-2116-4f59-b84c-93060dd44d08-registry-tls\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.350007 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.350016    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4354a2be-2116-4f59-b84c-93060dd44d08-bound-sa-token\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.350431 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.350177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4354a2be-2116-4f59-b84c-93060dd44d08-installation-pull-secrets\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.350431 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.350221    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4354a2be-2116-4f59-b84c-93060dd44d08-image-registry-private-configuration\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.350431 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.350254    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4354a2be-2116-4f59-b84c-93060dd44d08-ca-trust-extracted\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.350431 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.350287    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4354a2be-2116-4f59-b84c-93060dd44d08-trusted-ca\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.350431 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.350308    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4354a2be-2116-4f59-b84c-93060dd44d08-registry-certificates\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.350431 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.350409    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjsf2\" (UniqueName: \"kubernetes.io/projected/4354a2be-2116-4f59-b84c-93060dd44d08-kube-api-access-wjsf2\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.350798 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.350776    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4354a2be-2116-4f59-b84c-93060dd44d08-ca-trust-extracted\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.351258 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.351231    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4354a2be-2116-4f59-b84c-93060dd44d08-trusted-ca\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.351554 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.351528    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4354a2be-2116-4f59-b84c-93060dd44d08-registry-certificates\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.352842 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.352805    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4354a2be-2116-4f59-b84c-93060dd44d08-image-registry-private-configuration\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.352944 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.352882    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4354a2be-2116-4f59-b84c-93060dd44d08-installation-pull-secrets\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.352985 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.352975    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4354a2be-2116-4f59-b84c-93060dd44d08-registry-tls\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.382081 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.382013    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4354a2be-2116-4f59-b84c-93060dd44d08-bound-sa-token\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.386832 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.386804    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjsf2\" (UniqueName: \"kubernetes.io/projected/4354a2be-2116-4f59-b84c-93060dd44d08-kube-api-access-wjsf2\") pod \"image-registry-68f546c58-hqpr7\" (UID: \"4354a2be-2116-4f59-b84c-93060dd44d08\") " pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.403555 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.403527    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-56dzj"]
Sep 16 18:41:06.406033 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:41:06.406005    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84d2a1b_19d2_41cf_8ff2_510c1e37705c.slice/crio-395163b8ef6742b42393e4a417b2750fd582c3632497314658b79cb7f6f4cc4b WatchSource:0}: Error finding container 395163b8ef6742b42393e4a417b2750fd582c3632497314658b79cb7f6f4cc4b: Status 404 returned error can't find the container with id 395163b8ef6742b42393e4a417b2750fd582c3632497314658b79cb7f6f4cc4b
Sep 16 18:41:06.435995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.435954    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:06.618065 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.618023    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ww4bs" event={"ID":"d1a62b24-aeeb-43c1-8279-3d4e479eb481","Type":"ContainerStarted","Data":"7ae0ecd095f9449b04f2d86c692c49926381796e5ada007c7f0b5fa1e9c3147e"}
Sep 16 18:41:06.619238 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.619212    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ts6zv" event={"ID":"5acc4528-30cd-4e26-a485-02ba031ce861","Type":"ContainerStarted","Data":"b37603555f0d26d574a83590d9724794d620bc54d9b6f985c7e7ffea2da07845"}
Sep 16 18:41:06.620312 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.620293    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-56dzj" event={"ID":"d84d2a1b-19d2-41cf-8ff2-510c1e37705c","Type":"ContainerStarted","Data":"395163b8ef6742b42393e4a417b2750fd582c3632497314658b79cb7f6f4cc4b"}
Sep 16 18:41:06.788118 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:06.788078    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68f546c58-hqpr7"]
Sep 16 18:41:06.792031 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:41:06.791994    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4354a2be_2116_4f59_b84c_93060dd44d08.slice/crio-97939c36e855306a9f4ac2b71ac5d820437c7878be876510e162edda9230bcec WatchSource:0}: Error finding container 97939c36e855306a9f4ac2b71ac5d820437c7878be876510e162edda9230bcec: Status 404 returned error can't find the container with id 97939c36e855306a9f4ac2b71ac5d820437c7878be876510e162edda9230bcec
Sep 16 18:41:07.625575 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:07.625536    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68f546c58-hqpr7" event={"ID":"4354a2be-2116-4f59-b84c-93060dd44d08","Type":"ContainerStarted","Data":"7cde194ede9a71a1b7d5d57175326e851da3631483cd2932f34ec7a3130c8171"}
Sep 16 18:41:07.625750 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:07.625590    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68f546c58-hqpr7" event={"ID":"4354a2be-2116-4f59-b84c-93060dd44d08","Type":"ContainerStarted","Data":"97939c36e855306a9f4ac2b71ac5d820437c7878be876510e162edda9230bcec"}
Sep 16 18:41:07.625750 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:07.625648    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:07.691299 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:07.691237    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68f546c58-hqpr7" podStartSLOduration=1.6912137280000001 podStartE2EDuration="1.691213728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:07.690895709 +0000 UTC m=+106.050413716" watchObservedRunningTime="2026-09-16 18:41:07.691213728 +0000 UTC m=+106.050731724"
Sep 16 18:41:08.631712 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:08.631667    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ww4bs" event={"ID":"d1a62b24-aeeb-43c1-8279-3d4e479eb481","Type":"ContainerStarted","Data":"eded1265e57c8f298824069d77d4caa57ead9f504058b953d015acea77f2e95b"}
Sep 16 18:41:08.633216 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:08.633191    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ts6zv" event={"ID":"5acc4528-30cd-4e26-a485-02ba031ce861","Type":"ContainerStarted","Data":"bfc24379c1deb0c0ea3f878e42b2cbc8b5b53a7dd5e52334599ab7ec1884b0d1"}
Sep 16 18:41:08.634819 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:08.634782    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-56dzj" event={"ID":"d84d2a1b-19d2-41cf-8ff2-510c1e37705c","Type":"ContainerStarted","Data":"f973e04010aed1d25f1bad27c93d2a8b289af5835fee23641efb2c067b5a3c93"}
Sep 16 18:41:08.634945 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:08.634860    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-56dzj"
Sep 16 18:41:08.639692 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:08.639665    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-56dzj"
Sep 16 18:41:08.660585 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:08.660525    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ww4bs" podStartSLOduration=33.48347824 podStartE2EDuration="36.660507907s" totalImagesPullingTime="3.177029667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:08.659856741 +0000 UTC m=+107.019374762" watchObservedRunningTime="2026-09-16 18:41:08.660507907 +0000 UTC m=+107.020025901"
Sep 16 18:41:08.741977 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:08.741918    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ts6zv" podStartSLOduration=1.453061179 podStartE2EDuration="3.741895975s" totalImagesPullingTime="2.288834796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:08.712597493 +0000 UTC m=+107.072115488" watchObservedRunningTime="2026-09-16 18:41:08.741895975 +0000 UTC m=+107.101413970"
Sep 16 18:41:08.742454 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:08.742427    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-56dzj" podStartSLOduration=1.663240213 podStartE2EDuration="3.742415872s" totalImagesPullingTime="2.079175659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:08.741161158 +0000 UTC m=+107.100679154" watchObservedRunningTime="2026-09-16 18:41:08.742415872 +0000 UTC m=+107.101933866"
Sep 16 18:41:13.610667 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:13.610635    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x2brk"
Sep 16 18:41:14.689332 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.689298    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65"]
Sep 16 18:41:14.694682 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.694662    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65"
Sep 16 18:41:14.701166 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.700781    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 18:41:14.707535 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.707508    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-p2f5n\""
Sep 16 18:41:14.707790 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.707765    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:41:14.707875 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.707771    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:41:14.708025 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.708012    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:41:14.708078 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.708062    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:41:14.796520 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.796481    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65"]
Sep 16 18:41:14.819525 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.819486    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32481078-8fdf-47e9-b0f6-910736b236cf-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-jgd65\" (UID: \"32481078-8fdf-47e9-b0f6-910736b236cf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65"
Sep 16 18:41:14.819694 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.819533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7mp5\" (UniqueName: \"kubernetes.io/projected/32481078-8fdf-47e9-b0f6-910736b236cf-kube-api-access-d7mp5\") pod \"openshift-state-metrics-67dcf8c979-jgd65\" (UID: \"32481078-8fdf-47e9-b0f6-910736b236cf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65"
Sep 16 18:41:14.819694 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.819578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32481078-8fdf-47e9-b0f6-910736b236cf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-jgd65\" (UID: \"32481078-8fdf-47e9-b0f6-910736b236cf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65"
Sep 16 18:41:14.819771 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.819665    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/32481078-8fdf-47e9-b0f6-910736b236cf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-jgd65\" (UID: \"32481078-8fdf-47e9-b0f6-910736b236cf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65"
Sep 16 18:41:14.830059 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.830028    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-958gr"]
Sep 16 18:41:14.833002 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.832985    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:14.840758 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.840738    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:41:14.841106 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.841091    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:41:14.841487 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.841455    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:41:14.841577 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.841562    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cxq8v\""
Sep 16 18:41:14.920934 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.920893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-node-exporter-accelerators-collector-config\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:14.921116 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.920942    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32481078-8fdf-47e9-b0f6-910736b236cf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-jgd65\" (UID: \"32481078-8fdf-47e9-b0f6-910736b236cf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65"
Sep 16 18:41:14.921116 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.921002    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-node-exporter-tls\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:14.921116 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.921041    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:14.921116 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.921078    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-metrics-client-ca\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:14.921116 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.921107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw5hs\" (UniqueName: \"kubernetes.io/projected/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-kube-api-access-zw5hs\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:14.921302 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.921173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-root\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:14.921302 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.921217    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/32481078-8fdf-47e9-b0f6-910736b236cf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-jgd65\" (UID: \"32481078-8fdf-47e9-b0f6-910736b236cf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65"
Sep 16 18:41:14.921302 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.921241    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32481078-8fdf-47e9-b0f6-910736b236cf-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-jgd65\" (UID: \"32481078-8fdf-47e9-b0f6-910736b236cf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65"
Sep 16 18:41:14.921302 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.921257    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-node-exporter-textfile\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:14.921302 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.921277    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-node-exporter-wtmp\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:14.921518 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.921328    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-sys\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:14.921518 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.921396    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7mp5\" (UniqueName: \"kubernetes.io/projected/32481078-8fdf-47e9-b0f6-910736b236cf-kube-api-access-d7mp5\") pod \"openshift-state-metrics-67dcf8c979-jgd65\" (UID: \"32481078-8fdf-47e9-b0f6-910736b236cf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65"
Sep 16 18:41:14.921893 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.921870    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32481078-8fdf-47e9-b0f6-910736b236cf-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-jgd65\" (UID: \"32481078-8fdf-47e9-b0f6-910736b236cf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65"
Sep 16 18:41:14.923679 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.923652    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32481078-8fdf-47e9-b0f6-910736b236cf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-jgd65\" (UID: \"32481078-8fdf-47e9-b0f6-910736b236cf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65"
Sep 16 18:41:14.923763 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.923664    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/32481078-8fdf-47e9-b0f6-910736b236cf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-jgd65\" (UID: \"32481078-8fdf-47e9-b0f6-910736b236cf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65"
Sep 16 18:41:14.963375 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:14.963292    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7mp5\" (UniqueName: \"kubernetes.io/projected/32481078-8fdf-47e9-b0f6-910736b236cf-kube-api-access-d7mp5\") pod \"openshift-state-metrics-67dcf8c979-jgd65\" (UID: \"32481078-8fdf-47e9-b0f6-910736b236cf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65"
Sep 16 18:41:15.003897 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.003860    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65"
Sep 16 18:41:15.022072 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.022040    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-metrics-client-ca\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:15.022387 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.022079    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw5hs\" (UniqueName: \"kubernetes.io/projected/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-kube-api-access-zw5hs\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:15.022387 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.022115    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-root\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:15.022387 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.022164    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-node-exporter-textfile\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:15.022387 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.022189    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-node-exporter-wtmp\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:15.022387 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.022220    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-sys\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:15.022387 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.022261    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-node-exporter-accelerators-collector-config\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:15.022387 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.022316    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-node-exporter-tls\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:15.022387 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.022325    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-sys\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:15.022387 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.022356    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:15.022885 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.022259    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-root\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:15.022885 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.022690    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-node-exporter-wtmp\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:15.022885 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.022763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-metrics-client-ca\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:15.022885 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.022800    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-node-exporter-textfile\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:15.022885 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.022838    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-node-exporter-accelerators-collector-config\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:15.024971 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.024950    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-node-exporter-tls\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:15.025058 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.025015    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:15.038403 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.038361    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw5hs\" (UniqueName: \"kubernetes.io/projected/05cb61a7-1fb8-494f-a7f7-ad4c4808789f-kube-api-access-zw5hs\") pod \"node-exporter-958gr\" (UID: \"05cb61a7-1fb8-494f-a7f7-ad4c4808789f\") " pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:15.142027 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.141990    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-958gr"
Sep 16 18:41:15.153893 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:41:15.153849    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05cb61a7_1fb8_494f_a7f7_ad4c4808789f.slice/crio-742ff300b95904cfedb49c77e06fc716108d1ac7e387b2d4c644d6b1aed6e663 WatchSource:0}: Error finding container 742ff300b95904cfedb49c77e06fc716108d1ac7e387b2d4c644d6b1aed6e663: Status 404 returned error can't find the container with id 742ff300b95904cfedb49c77e06fc716108d1ac7e387b2d4c644d6b1aed6e663
Sep 16 18:41:15.306024 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.305970    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65"]
Sep 16 18:41:15.308551 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:41:15.308527    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32481078_8fdf_47e9_b0f6_910736b236cf.slice/crio-f3a43a9bb0a2ef44be46e283e9c88c968c2cbbdc053553c98137991303ed2fca WatchSource:0}: Error finding container f3a43a9bb0a2ef44be46e283e9c88c968c2cbbdc053553c98137991303ed2fca: Status 404 returned error can't find the container with id f3a43a9bb0a2ef44be46e283e9c88c968c2cbbdc053553c98137991303ed2fca
Sep 16 18:41:15.660540 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.660457    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65" event={"ID":"32481078-8fdf-47e9-b0f6-910736b236cf","Type":"ContainerStarted","Data":"4f1d219ffc76b889d22b0c30e6706762d864f7f13964fd16c091dc02d46e4743"}
Sep 16 18:41:15.660701 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.660538    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65" event={"ID":"32481078-8fdf-47e9-b0f6-910736b236cf","Type":"ContainerStarted","Data":"738f6df7d2e1cb545cecc08261dbbb8afe8023cf349cdd895ffd784ae2cef73b"}
Sep 16 18:41:15.660701 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.660575    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65" event={"ID":"32481078-8fdf-47e9-b0f6-910736b236cf","Type":"ContainerStarted","Data":"f3a43a9bb0a2ef44be46e283e9c88c968c2cbbdc053553c98137991303ed2fca"}
Sep 16 18:41:15.661982 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:15.661937    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-958gr" event={"ID":"05cb61a7-1fb8-494f-a7f7-ad4c4808789f","Type":"ContainerStarted","Data":"742ff300b95904cfedb49c77e06fc716108d1ac7e387b2d4c644d6b1aed6e663"}
Sep 16 18:41:16.669783 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:16.669746    2565 generic.go:356] "Generic (PLEG): container finished" podID="05cb61a7-1fb8-494f-a7f7-ad4c4808789f" containerID="a6b440ed95d0634bd28567f528062a107acd214d1f567c4243853d840a9b5a37" exitCode=0
Sep 16 18:41:16.670221 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:16.669839    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-958gr" event={"ID":"05cb61a7-1fb8-494f-a7f7-ad4c4808789f","Type":"ContainerDied","Data":"a6b440ed95d0634bd28567f528062a107acd214d1f567c4243853d840a9b5a37"}
Sep 16 18:41:16.672163 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:16.672131    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65" event={"ID":"32481078-8fdf-47e9-b0f6-910736b236cf","Type":"ContainerStarted","Data":"b54c3ca4812be0c421a4a250e292654d6f494738c681b70687d838dddeea0114"}
Sep 16 18:41:16.824084 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:16.824024    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-jgd65" podStartSLOduration=1.782635184 podStartE2EDuration="2.824006302s" totalImagesPullingTime="1.041371118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:16.823565956 +0000 UTC m=+115.183083950" watchObservedRunningTime="2026-09-16 18:41:16.824006302 +0000 UTC m=+115.183524291"
Sep 16 18:41:17.677982 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:17.677942    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-958gr" event={"ID":"05cb61a7-1fb8-494f-a7f7-ad4c4808789f","Type":"ContainerStarted","Data":"73f88a0c51b1a4bb8475eccba022d365e5bd7204e3a87adddc358f2537ec5764"}
Sep 16 18:41:17.678449 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:17.678012    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-958gr" event={"ID":"05cb61a7-1fb8-494f-a7f7-ad4c4808789f","Type":"ContainerStarted","Data":"7dbea267b9d53511d28826544249b694cbbab4d8dc603583e40a68b6c1a270f3"}
Sep 16 18:41:17.728474 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:17.728414    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-958gr" podStartSLOduration=2.632530609 podStartE2EDuration="3.728390552s" totalImagesPullingTime="1.023343184s" totalInitContainerRuntime="72.516759ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:17.72816074 +0000 UTC m=+116.087678749" watchObservedRunningTime="2026-09-16 18:41:17.728390552 +0000 UTC m=+116.087908547"
Sep 16 18:41:20.209187 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.209150    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-69dd5589d6-tk55g"]
Sep 16 18:41:20.212612 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.212593    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.216378 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.216354    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 18:41:20.216859 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.216840    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 18:41:20.216946 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.216925    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 18:41:20.217004 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.216983    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 18:41:20.217059 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.216995    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 18:41:20.221199 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.221179    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4kknv\""
Sep 16 18:41:20.231197 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.231167    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69dd5589d6-tk55g"]
Sep 16 18:41:20.248398 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.248335    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 18:41:20.367423 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.367338    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76b8bb64-3493-452f-9a4d-f1cc378be276-metrics-client-ca\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.367423 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.367430    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/76b8bb64-3493-452f-9a4d-f1cc378be276-telemeter-client-tls\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.367655 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.367450    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2tpz\" (UniqueName: \"kubernetes.io/projected/76b8bb64-3493-452f-9a4d-f1cc378be276-kube-api-access-q2tpz\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.367655 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.367502    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76b8bb64-3493-452f-9a4d-f1cc378be276-serving-certs-ca-bundle\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.367655 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.367538    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76b8bb64-3493-452f-9a4d-f1cc378be276-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.367655 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.367609    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/76b8bb64-3493-452f-9a4d-f1cc378be276-secret-telemeter-client\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.367816 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.367670    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/76b8bb64-3493-452f-9a4d-f1cc378be276-federate-client-tls\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.367816 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.367698    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76b8bb64-3493-452f-9a4d-f1cc378be276-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.468375 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.468257    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/76b8bb64-3493-452f-9a4d-f1cc378be276-federate-client-tls\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.468375 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.468301    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76b8bb64-3493-452f-9a4d-f1cc378be276-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.468375 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.468325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76b8bb64-3493-452f-9a4d-f1cc378be276-metrics-client-ca\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.468375 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.468361    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/76b8bb64-3493-452f-9a4d-f1cc378be276-telemeter-client-tls\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.468719 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.468385    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2tpz\" (UniqueName: \"kubernetes.io/projected/76b8bb64-3493-452f-9a4d-f1cc378be276-kube-api-access-q2tpz\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.468719 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.468430    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76b8bb64-3493-452f-9a4d-f1cc378be276-serving-certs-ca-bundle\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.468719 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.468474    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76b8bb64-3493-452f-9a4d-f1cc378be276-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.468719 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.468512    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/76b8bb64-3493-452f-9a4d-f1cc378be276-secret-telemeter-client\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.469194 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.469165    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76b8bb64-3493-452f-9a4d-f1cc378be276-metrics-client-ca\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.469389 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.469314    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76b8bb64-3493-452f-9a4d-f1cc378be276-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.469389 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.469286    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76b8bb64-3493-452f-9a4d-f1cc378be276-serving-certs-ca-bundle\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.471087 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.471061    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/76b8bb64-3493-452f-9a4d-f1cc378be276-federate-client-tls\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.471211 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.471125    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/76b8bb64-3493-452f-9a4d-f1cc378be276-telemeter-client-tls\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.471211 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.471181    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76b8bb64-3493-452f-9a4d-f1cc378be276-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.471308 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.471248    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/76b8bb64-3493-452f-9a4d-f1cc378be276-secret-telemeter-client\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.485387 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.485340    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2tpz\" (UniqueName: \"kubernetes.io/projected/76b8bb64-3493-452f-9a4d-f1cc378be276-kube-api-access-q2tpz\") pod \"telemeter-client-69dd5589d6-tk55g\" (UID: \"76b8bb64-3493-452f-9a4d-f1cc378be276\") " pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.522059 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.522017    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g"
Sep 16 18:41:20.820679 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:20.820521    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69dd5589d6-tk55g"]
Sep 16 18:41:20.823291 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:41:20.823248    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b8bb64_3493_452f_9a4d_f1cc378be276.slice/crio-d1d1ddfc6aefd5cf2d81c7bea2458d8d61dc11e2a8deb2f561602636a136944f WatchSource:0}: Error finding container d1d1ddfc6aefd5cf2d81c7bea2458d8d61dc11e2a8deb2f561602636a136944f: Status 404 returned error can't find the container with id d1d1ddfc6aefd5cf2d81c7bea2458d8d61dc11e2a8deb2f561602636a136944f
Sep 16 18:41:21.695398 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:21.695355    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g" event={"ID":"76b8bb64-3493-452f-9a4d-f1cc378be276","Type":"ContainerStarted","Data":"d1d1ddfc6aefd5cf2d81c7bea2458d8d61dc11e2a8deb2f561602636a136944f"}
Sep 16 18:41:23.027234 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:23.027148    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-sbbgn"]
Sep 16 18:41:23.030441 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:23.030422    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-sbbgn"
Sep 16 18:41:23.034049 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:23.034014    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:41:23.034521 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:23.034500    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-h9nx9\""
Sep 16 18:41:23.034847 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:23.034831    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:41:23.067100 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:23.067064    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-sbbgn"]
Sep 16 18:41:23.192205 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:23.192166    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf7cw\" (UniqueName: \"kubernetes.io/projected/a4d4d25d-c795-416f-8398-d136b174cb36-kube-api-access-rf7cw\") pod \"downloads-76f99584f4-sbbgn\" (UID: \"a4d4d25d-c795-416f-8398-d136b174cb36\") " pod="openshift-console/downloads-76f99584f4-sbbgn"
Sep 16 18:41:23.293459 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:23.293367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf7cw\" (UniqueName: \"kubernetes.io/projected/a4d4d25d-c795-416f-8398-d136b174cb36-kube-api-access-rf7cw\") pod \"downloads-76f99584f4-sbbgn\" (UID: \"a4d4d25d-c795-416f-8398-d136b174cb36\") " pod="openshift-console/downloads-76f99584f4-sbbgn"
Sep 16 18:41:23.322036 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:23.321997    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf7cw\" (UniqueName: \"kubernetes.io/projected/a4d4d25d-c795-416f-8398-d136b174cb36-kube-api-access-rf7cw\") pod \"downloads-76f99584f4-sbbgn\" (UID: \"a4d4d25d-c795-416f-8398-d136b174cb36\") " pod="openshift-console/downloads-76f99584f4-sbbgn"
Sep 16 18:41:23.339770 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:23.339733    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-sbbgn"
Sep 16 18:41:23.704936 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:23.704847    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g" event={"ID":"76b8bb64-3493-452f-9a4d-f1cc378be276","Type":"ContainerStarted","Data":"3a7ef683d5217b3a9be28678c98f5b86476f49e2e99340a35fd449b84366eb64"}
Sep 16 18:41:23.728767 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:23.728736    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-sbbgn"]
Sep 16 18:41:23.822307 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:41:23.822270    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d4d25d_c795_416f_8398_d136b174cb36.slice/crio-5266d75da55f3afabeffe0ae99000575ae6c4b11765202dde5eaa528475455c8 WatchSource:0}: Error finding container 5266d75da55f3afabeffe0ae99000575ae6c4b11765202dde5eaa528475455c8: Status 404 returned error can't find the container with id 5266d75da55f3afabeffe0ae99000575ae6c4b11765202dde5eaa528475455c8
Sep 16 18:41:24.714160 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:24.714114    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g" event={"ID":"76b8bb64-3493-452f-9a4d-f1cc378be276","Type":"ContainerStarted","Data":"6a2f337f16e9b2f81970e76ad9ff4ddbe9183a4fa4b948b936038f867ca0c8f4"}
Sep 16 18:41:24.714657 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:24.714197    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g" event={"ID":"76b8bb64-3493-452f-9a4d-f1cc378be276","Type":"ContainerStarted","Data":"d78d447fbbc488f01c9a90d5926232d92de9713610fe9adfe0b16be78037890e"}
Sep 16 18:41:24.715741 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:24.715713    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-sbbgn" event={"ID":"a4d4d25d-c795-416f-8398-d136b174cb36","Type":"ContainerStarted","Data":"5266d75da55f3afabeffe0ae99000575ae6c4b11765202dde5eaa528475455c8"}
Sep 16 18:41:24.774018 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:24.773957    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-69dd5589d6-tk55g" podStartSLOduration=1.809420595 podStartE2EDuration="4.77393815s" totalImagesPullingTime="2.964517555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:24.772633827 +0000 UTC m=+123.132151823" watchObservedRunningTime="2026-09-16 18:41:24.77393815 +0000 UTC m=+123.133456144"
Sep 16 18:41:25.732320 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:25.732287    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:41:28.640657 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:28.640619    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68f546c58-hqpr7"
Sep 16 18:41:30.745363 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:30.745308    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6" podUID="9ab9989d-a434-4aa6-a97a-16410c87c1af" containerName="registry" containerID="cri-o://3efd8ed006bd2a95a5db00483002efc298d33d1a4b6a38681926a6f2da7e6f5a" gracePeriod=30
Sep 16 18:41:31.024886 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.024857    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:41:31.169324 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.169271    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ab9989d-a434-4aa6-a97a-16410c87c1af-installation-pull-secrets\") pod \"9ab9989d-a434-4aa6-a97a-16410c87c1af\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") "
Sep 16 18:41:31.169535 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.169359    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ab9989d-a434-4aa6-a97a-16410c87c1af-image-registry-private-configuration\") pod \"9ab9989d-a434-4aa6-a97a-16410c87c1af\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") "
Sep 16 18:41:31.169535 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.169396    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8g6pf\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-kube-api-access-8g6pf\") pod \"9ab9989d-a434-4aa6-a97a-16410c87c1af\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") "
Sep 16 18:41:31.169535 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.169422    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-certificates\") pod \"9ab9989d-a434-4aa6-a97a-16410c87c1af\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") "
Sep 16 18:41:31.169535 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.169457    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls\") pod \"9ab9989d-a434-4aa6-a97a-16410c87c1af\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") "
Sep 16 18:41:31.169535 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.169487    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-bound-sa-token\") pod \"9ab9989d-a434-4aa6-a97a-16410c87c1af\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") "
Sep 16 18:41:31.169535 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.169527    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ab9989d-a434-4aa6-a97a-16410c87c1af-trusted-ca\") pod \"9ab9989d-a434-4aa6-a97a-16410c87c1af\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") "
Sep 16 18:41:31.169849 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.169566    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ab9989d-a434-4aa6-a97a-16410c87c1af-ca-trust-extracted\") pod \"9ab9989d-a434-4aa6-a97a-16410c87c1af\" (UID: \"9ab9989d-a434-4aa6-a97a-16410c87c1af\") "
Sep 16 18:41:31.170687 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.170630    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9ab9989d-a434-4aa6-a97a-16410c87c1af" (UID: "9ab9989d-a434-4aa6-a97a-16410c87c1af"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:41:31.170845 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.170642    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab9989d-a434-4aa6-a97a-16410c87c1af-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9ab9989d-a434-4aa6-a97a-16410c87c1af" (UID: "9ab9989d-a434-4aa6-a97a-16410c87c1af"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:41:31.172680 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.172612    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ab9989d-a434-4aa6-a97a-16410c87c1af-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9ab9989d-a434-4aa6-a97a-16410c87c1af" (UID: "9ab9989d-a434-4aa6-a97a-16410c87c1af"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:41:31.172967 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.172944    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9ab9989d-a434-4aa6-a97a-16410c87c1af" (UID: "9ab9989d-a434-4aa6-a97a-16410c87c1af"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:41:31.173042 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.172985    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9ab9989d-a434-4aa6-a97a-16410c87c1af" (UID: "9ab9989d-a434-4aa6-a97a-16410c87c1af"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:41:31.173119 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.173087    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-kube-api-access-8g6pf" (OuterVolumeSpecName: "kube-api-access-8g6pf") pod "9ab9989d-a434-4aa6-a97a-16410c87c1af" (UID: "9ab9989d-a434-4aa6-a97a-16410c87c1af"). InnerVolumeSpecName "kube-api-access-8g6pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:41:31.173797 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.173761    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ab9989d-a434-4aa6-a97a-16410c87c1af-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9ab9989d-a434-4aa6-a97a-16410c87c1af" (UID: "9ab9989d-a434-4aa6-a97a-16410c87c1af"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:41:31.181707 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.181674    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab9989d-a434-4aa6-a97a-16410c87c1af-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9ab9989d-a434-4aa6-a97a-16410c87c1af" (UID: "9ab9989d-a434-4aa6-a97a-16410c87c1af"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:31.271161 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.271060    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ab9989d-a434-4aa6-a97a-16410c87c1af-trusted-ca\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:41:31.271161 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.271097    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ab9989d-a434-4aa6-a97a-16410c87c1af-ca-trust-extracted\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:41:31.271161 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.271112    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ab9989d-a434-4aa6-a97a-16410c87c1af-installation-pull-secrets\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:41:31.271161 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.271126    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ab9989d-a434-4aa6-a97a-16410c87c1af-image-registry-private-configuration\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:41:31.271161 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.271138    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g6pf\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-kube-api-access-8g6pf\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:41:31.271161 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.271150    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-certificates\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:41:31.271161 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.271163    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-registry-tls\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:41:31.271529 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.271175    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ab9989d-a434-4aa6-a97a-16410c87c1af-bound-sa-token\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:41:31.749290 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.749251    2565 generic.go:356] "Generic (PLEG): container finished" podID="9ab9989d-a434-4aa6-a97a-16410c87c1af" containerID="3efd8ed006bd2a95a5db00483002efc298d33d1a4b6a38681926a6f2da7e6f5a" exitCode=0
Sep 16 18:41:31.749754 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.749304    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"
Sep 16 18:41:31.749754 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.749316    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6" event={"ID":"9ab9989d-a434-4aa6-a97a-16410c87c1af","Type":"ContainerDied","Data":"3efd8ed006bd2a95a5db00483002efc298d33d1a4b6a38681926a6f2da7e6f5a"}
Sep 16 18:41:31.749754 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.749386    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d74c7ddcb-wg6c6" event={"ID":"9ab9989d-a434-4aa6-a97a-16410c87c1af","Type":"ContainerDied","Data":"cca7e9df8173f15033164c9fa08ebca4d43893e4b094b557f9146ffca7f9da54"}
Sep 16 18:41:31.749754 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.749418    2565 scope.go:117] "RemoveContainer" containerID="3efd8ed006bd2a95a5db00483002efc298d33d1a4b6a38681926a6f2da7e6f5a"
Sep 16 18:41:31.760037 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.760012    2565 scope.go:117] "RemoveContainer" containerID="3efd8ed006bd2a95a5db00483002efc298d33d1a4b6a38681926a6f2da7e6f5a"
Sep 16 18:41:31.760438 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:41:31.760339    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3efd8ed006bd2a95a5db00483002efc298d33d1a4b6a38681926a6f2da7e6f5a\": container with ID starting with 3efd8ed006bd2a95a5db00483002efc298d33d1a4b6a38681926a6f2da7e6f5a not found: ID does not exist" containerID="3efd8ed006bd2a95a5db00483002efc298d33d1a4b6a38681926a6f2da7e6f5a"
Sep 16 18:41:31.760515 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.760453    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3efd8ed006bd2a95a5db00483002efc298d33d1a4b6a38681926a6f2da7e6f5a"} err="failed to get container status \"3efd8ed006bd2a95a5db00483002efc298d33d1a4b6a38681926a6f2da7e6f5a\": rpc error: code = NotFound desc = could not find container \"3efd8ed006bd2a95a5db00483002efc298d33d1a4b6a38681926a6f2da7e6f5a\": container with ID starting with 3efd8ed006bd2a95a5db00483002efc298d33d1a4b6a38681926a6f2da7e6f5a not found: ID does not exist"
Sep 16 18:41:31.785269 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.785235    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"]
Sep 16 18:41:31.792921 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.792890    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7d74c7ddcb-wg6c6"]
Sep 16 18:41:31.875297 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.875261    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs\") pod \"network-metrics-daemon-7r6xr\" (UID: \"2344014b-38cc-448f-9c9d-6b49d8e80b85\") " pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:41:31.877998 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:31.877967    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2344014b-38cc-448f-9c9d-6b49d8e80b85-metrics-certs\") pod \"network-metrics-daemon-7r6xr\" (UID: \"2344014b-38cc-448f-9c9d-6b49d8e80b85\") " pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:41:32.128478 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.128392    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vcswh\""
Sep 16 18:41:32.132496 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.132465    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7r6xr"
Sep 16 18:41:32.209141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.208993    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ab9989d-a434-4aa6-a97a-16410c87c1af" path="/var/lib/kubelet/pods/9ab9989d-a434-4aa6-a97a-16410c87c1af/volumes"
Sep 16 18:41:32.466826 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:41:32.466788    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2344014b_38cc_448f_9c9d_6b49d8e80b85.slice/crio-7002a25104fe6ae1e3ad72634d77d050620f8a24663b05f78b29fd1d39b8c568 WatchSource:0}: Error finding container 7002a25104fe6ae1e3ad72634d77d050620f8a24663b05f78b29fd1d39b8c568: Status 404 returned error can't find the container with id 7002a25104fe6ae1e3ad72634d77d050620f8a24663b05f78b29fd1d39b8c568
Sep 16 18:41:32.474838 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.470586    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7r6xr"]
Sep 16 18:41:32.755243 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.755210    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5754bd5445-lvdtm"]
Sep 16 18:41:32.755744 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.755616    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ab9989d-a434-4aa6-a97a-16410c87c1af" containerName="registry"
Sep 16 18:41:32.755744 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.755635    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ab9989d-a434-4aa6-a97a-16410c87c1af" containerName="registry"
Sep 16 18:41:32.755744 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.755702    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ab9989d-a434-4aa6-a97a-16410c87c1af" containerName="registry"
Sep 16 18:41:32.760146 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.760120    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7r6xr" event={"ID":"2344014b-38cc-448f-9c9d-6b49d8e80b85","Type":"ContainerStarted","Data":"7002a25104fe6ae1e3ad72634d77d050620f8a24663b05f78b29fd1d39b8c568"}
Sep 16 18:41:32.760259 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.760215    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:32.765385 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.765338    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:41:32.765540 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.765338    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bct8d\""
Sep 16 18:41:32.775812 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.775788    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:41:32.775957 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.775851    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:41:32.775957 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.775851    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:41:32.783332 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.783299    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a04c85b4-ddee-4abd-982f-0d84f23b7926-oauth-serving-cert\") pod \"console-5754bd5445-lvdtm\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") " pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:32.783515 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.783366    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a04c85b4-ddee-4abd-982f-0d84f23b7926-console-config\") pod \"console-5754bd5445-lvdtm\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") " pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:32.783515 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.783394    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a04c85b4-ddee-4abd-982f-0d84f23b7926-service-ca\") pod \"console-5754bd5445-lvdtm\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") " pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:32.783515 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.783496    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a04c85b4-ddee-4abd-982f-0d84f23b7926-console-serving-cert\") pod \"console-5754bd5445-lvdtm\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") " pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:32.783655 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.783531    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6jqj\" (UniqueName: \"kubernetes.io/projected/a04c85b4-ddee-4abd-982f-0d84f23b7926-kube-api-access-j6jqj\") pod \"console-5754bd5445-lvdtm\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") " pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:32.783655 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.783573    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a04c85b4-ddee-4abd-982f-0d84f23b7926-console-oauth-config\") pod \"console-5754bd5445-lvdtm\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") " pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:32.784111 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.784081    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:41:32.830659 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.830621    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5754bd5445-lvdtm"]
Sep 16 18:41:32.884503 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.884460    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a04c85b4-ddee-4abd-982f-0d84f23b7926-console-oauth-config\") pod \"console-5754bd5445-lvdtm\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") " pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:32.884690 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.884538    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a04c85b4-ddee-4abd-982f-0d84f23b7926-oauth-serving-cert\") pod \"console-5754bd5445-lvdtm\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") " pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:32.884690 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.884578    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a04c85b4-ddee-4abd-982f-0d84f23b7926-console-config\") pod \"console-5754bd5445-lvdtm\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") " pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:32.884690 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.884609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a04c85b4-ddee-4abd-982f-0d84f23b7926-service-ca\") pod \"console-5754bd5445-lvdtm\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") " pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:32.884690 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.884667    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a04c85b4-ddee-4abd-982f-0d84f23b7926-console-serving-cert\") pod \"console-5754bd5445-lvdtm\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") " pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:32.884910 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.884699    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6jqj\" (UniqueName: \"kubernetes.io/projected/a04c85b4-ddee-4abd-982f-0d84f23b7926-kube-api-access-j6jqj\") pod \"console-5754bd5445-lvdtm\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") " pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:32.885613 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.885579    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a04c85b4-ddee-4abd-982f-0d84f23b7926-console-config\") pod \"console-5754bd5445-lvdtm\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") " pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:32.885743 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.885616    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a04c85b4-ddee-4abd-982f-0d84f23b7926-service-ca\") pod \"console-5754bd5445-lvdtm\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") " pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:32.885743 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.885651    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a04c85b4-ddee-4abd-982f-0d84f23b7926-oauth-serving-cert\") pod \"console-5754bd5445-lvdtm\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") " pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:32.887770 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.887727    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a04c85b4-ddee-4abd-982f-0d84f23b7926-console-oauth-config\") pod \"console-5754bd5445-lvdtm\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") " pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:32.888192 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.888168    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a04c85b4-ddee-4abd-982f-0d84f23b7926-console-serving-cert\") pod \"console-5754bd5445-lvdtm\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") " pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:32.914555 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:32.914519    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6jqj\" (UniqueName: \"kubernetes.io/projected/a04c85b4-ddee-4abd-982f-0d84f23b7926-kube-api-access-j6jqj\") pod \"console-5754bd5445-lvdtm\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") " pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:33.074232 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:33.074143    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:41:33.395364 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:33.395322    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5754bd5445-lvdtm"]
Sep 16 18:41:33.401290 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:41:33.401258    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04c85b4_ddee_4abd_982f_0d84f23b7926.slice/crio-9fd689d5a0d04ecb4e5f0003b188eef3e1cfa0376e10c5e084e23440efbd475c WatchSource:0}: Error finding container 9fd689d5a0d04ecb4e5f0003b188eef3e1cfa0376e10c5e084e23440efbd475c: Status 404 returned error can't find the container with id 9fd689d5a0d04ecb4e5f0003b188eef3e1cfa0376e10c5e084e23440efbd475c
Sep 16 18:41:33.763459 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:33.763419    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5754bd5445-lvdtm" event={"ID":"a04c85b4-ddee-4abd-982f-0d84f23b7926","Type":"ContainerStarted","Data":"9fd689d5a0d04ecb4e5f0003b188eef3e1cfa0376e10c5e084e23440efbd475c"}
Sep 16 18:41:40.798397 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:40.798328    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7r6xr" event={"ID":"2344014b-38cc-448f-9c9d-6b49d8e80b85","Type":"ContainerStarted","Data":"1caf1fc53941270cfdacdea58ace63bae2370ae2fc122851dea4d2d83e7e4a9f"}
Sep 16 18:41:40.798923 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:40.798426    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7r6xr" event={"ID":"2344014b-38cc-448f-9c9d-6b49d8e80b85","Type":"ContainerStarted","Data":"8549c5e8c5f330cc73ddc2fccf82a3e7eaceb9e008beacdec3df5c45362db83b"}
Sep 16 18:41:40.801598 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:40.801560    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-sbbgn" event={"ID":"a4d4d25d-c795-416f-8398-d136b174cb36","Type":"ContainerStarted","Data":"48d29cc6139ad4dc37732449b129c28655e8a6c76cf63b22a4444b3a67ab57fc"}
Sep 16 18:41:40.801750 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:40.801628    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-sbbgn"
Sep 16 18:41:40.814248 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:40.814210    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-sbbgn"
Sep 16 18:41:40.846068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:40.845967    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7r6xr" podStartSLOduration=130.895864276 podStartE2EDuration="2m18.84594365s" totalImagesPullingTime="7.950079374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:40.84507296 +0000 UTC m=+139.204590967" watchObservedRunningTime="2026-09-16 18:41:40.84594365 +0000 UTC m=+139.205461647"
Sep 16 18:41:40.884326 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:40.884257    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-sbbgn" podStartSLOduration=2.237485019 podStartE2EDuration="18.884233824s" totalImagesPullingTime="16.646748805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:40.884024436 +0000 UTC m=+139.243542433" watchObservedRunningTime="2026-09-16 18:41:40.884233824 +0000 UTC m=+139.243751835"
Sep 16 18:41:41.467593 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.467550    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57db5fbc7f-pv6f7"]
Sep 16 18:41:41.481321 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.481047    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.493103 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.493071    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:41:41.503378 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.503239    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57db5fbc7f-pv6f7"]
Sep 16 18:41:41.563433 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.563389    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltcz7\" (UniqueName: \"kubernetes.io/projected/9547c8de-0189-4a45-b201-686ed43dfa29-kube-api-access-ltcz7\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.563647 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.563444    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-trusted-ca-bundle\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.563647 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.563535    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-console-config\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.563647 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.563565    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-service-ca\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.563647 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.563588    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-oauth-serving-cert\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.563647 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.563628    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9547c8de-0189-4a45-b201-686ed43dfa29-console-oauth-config\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.563888 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.563659    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9547c8de-0189-4a45-b201-686ed43dfa29-console-serving-cert\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.664795 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.664747    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-console-config\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.665016 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.664807    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-service-ca\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.665016 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.664837    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-oauth-serving-cert\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.665016 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.664889    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9547c8de-0189-4a45-b201-686ed43dfa29-console-oauth-config\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.665016 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.664917    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9547c8de-0189-4a45-b201-686ed43dfa29-console-serving-cert\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.665016 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.664987    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltcz7\" (UniqueName: \"kubernetes.io/projected/9547c8de-0189-4a45-b201-686ed43dfa29-kube-api-access-ltcz7\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.665016 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.665019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-trusted-ca-bundle\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.666554 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.665806    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-console-config\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.666554 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.666045    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-trusted-ca-bundle\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.666931 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.666883    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-service-ca\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.667582 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.667469    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-oauth-serving-cert\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.669394 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.669321    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9547c8de-0189-4a45-b201-686ed43dfa29-console-serving-cert\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.670032 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.669899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9547c8de-0189-4a45-b201-686ed43dfa29-console-oauth-config\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.679657 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.679208    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltcz7\" (UniqueName: \"kubernetes.io/projected/9547c8de-0189-4a45-b201-686ed43dfa29-kube-api-access-ltcz7\") pod \"console-57db5fbc7f-pv6f7\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") " pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:41.796637 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:41.796334    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:42.109690 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:42.109649    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57db5fbc7f-pv6f7"]
Sep 16 18:41:43.360550 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:41:43.360505    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9547c8de_0189_4a45_b201_686ed43dfa29.slice/crio-ea4204c6d4fd0cd158a952a4e9c96b6685ed69ba4ca66be59547430d73aa7e4e WatchSource:0}: Error finding container ea4204c6d4fd0cd158a952a4e9c96b6685ed69ba4ca66be59547430d73aa7e4e: Status 404 returned error can't find the container with id ea4204c6d4fd0cd158a952a4e9c96b6685ed69ba4ca66be59547430d73aa7e4e
Sep 16 18:41:43.820997 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:43.820957    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57db5fbc7f-pv6f7" event={"ID":"9547c8de-0189-4a45-b201-686ed43dfa29","Type":"ContainerStarted","Data":"ea4204c6d4fd0cd158a952a4e9c96b6685ed69ba4ca66be59547430d73aa7e4e"}
Sep 16 18:41:43.823278 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:43.823237    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5754bd5445-lvdtm" event={"ID":"a04c85b4-ddee-4abd-982f-0d84f23b7926","Type":"ContainerStarted","Data":"8a682a1c53d34c37d531a275aac7c09f397896ba1b62a53f32e87fe1d0104bea"}
Sep 16 18:41:43.856147 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:43.856076    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5754bd5445-lvdtm" podStartSLOduration=1.559415437 podStartE2EDuration="11.856044885s" totalImagesPullingTime="10.296629448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:43.854779692 +0000 UTC m=+142.214297689" watchObservedRunningTime="2026-09-16 18:41:43.856044885 +0000 UTC m=+142.215562912"
Sep 16 18:41:44.829683 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:44.829638    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57db5fbc7f-pv6f7" event={"ID":"9547c8de-0189-4a45-b201-686ed43dfa29","Type":"ContainerStarted","Data":"2e0bb0c25c691c96158c0f1f6345b5c270058d51f40fb86ae4e2dd05445bdcbe"}
Sep 16 18:41:44.856789 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:44.856728    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57db5fbc7f-pv6f7" podStartSLOduration=3.352025927 podStartE2EDuration="3.85670825s" totalImagesPullingTime="504.682323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:44.854977619 +0000 UTC m=+143.214495614" watchObservedRunningTime="2026-09-16 18:41:44.85670825 +0000 UTC m=+143.216226246"
Sep 16 18:41:51.797504 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:51.797459    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:51.798075 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:51.797546    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:51.802320 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:51.802293    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:51.864042 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:51.864012    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:41:51.999885 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:51.999845    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5754bd5445-lvdtm"]
Sep 16 18:41:53.074948 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:41:53.074911    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:42:17.019431 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.019290    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5754bd5445-lvdtm" podUID="a04c85b4-ddee-4abd-982f-0d84f23b7926" containerName="console" containerID="cri-o://8a682a1c53d34c37d531a275aac7c09f397896ba1b62a53f32e87fe1d0104bea" gracePeriod=15
Sep 16 18:42:17.348107 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.348068    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:42:17.403889 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.403851    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a04c85b4-ddee-4abd-982f-0d84f23b7926-console-config\") pod \"a04c85b4-ddee-4abd-982f-0d84f23b7926\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") "
Sep 16 18:42:17.404082 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.403942    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a04c85b4-ddee-4abd-982f-0d84f23b7926-service-ca\") pod \"a04c85b4-ddee-4abd-982f-0d84f23b7926\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") "
Sep 16 18:42:17.404082 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.404006    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a04c85b4-ddee-4abd-982f-0d84f23b7926-oauth-serving-cert\") pod \"a04c85b4-ddee-4abd-982f-0d84f23b7926\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") "
Sep 16 18:42:17.404082 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.404037    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a04c85b4-ddee-4abd-982f-0d84f23b7926-console-oauth-config\") pod \"a04c85b4-ddee-4abd-982f-0d84f23b7926\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") "
Sep 16 18:42:17.404082 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.404066    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a04c85b4-ddee-4abd-982f-0d84f23b7926-console-serving-cert\") pod \"a04c85b4-ddee-4abd-982f-0d84f23b7926\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") "
Sep 16 18:42:17.404292 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.404114    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6jqj\" (UniqueName: \"kubernetes.io/projected/a04c85b4-ddee-4abd-982f-0d84f23b7926-kube-api-access-j6jqj\") pod \"a04c85b4-ddee-4abd-982f-0d84f23b7926\" (UID: \"a04c85b4-ddee-4abd-982f-0d84f23b7926\") "
Sep 16 18:42:17.404292 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.404280    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a04c85b4-ddee-4abd-982f-0d84f23b7926-console-config" (OuterVolumeSpecName: "console-config") pod "a04c85b4-ddee-4abd-982f-0d84f23b7926" (UID: "a04c85b4-ddee-4abd-982f-0d84f23b7926"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:17.404407 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.404372    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a04c85b4-ddee-4abd-982f-0d84f23b7926-console-config\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:42:17.404587 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.404549    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a04c85b4-ddee-4abd-982f-0d84f23b7926-service-ca" (OuterVolumeSpecName: "service-ca") pod "a04c85b4-ddee-4abd-982f-0d84f23b7926" (UID: "a04c85b4-ddee-4abd-982f-0d84f23b7926"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:17.404687 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.404649    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a04c85b4-ddee-4abd-982f-0d84f23b7926-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a04c85b4-ddee-4abd-982f-0d84f23b7926" (UID: "a04c85b4-ddee-4abd-982f-0d84f23b7926"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:17.406653 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.406625    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a04c85b4-ddee-4abd-982f-0d84f23b7926-kube-api-access-j6jqj" (OuterVolumeSpecName: "kube-api-access-j6jqj") pod "a04c85b4-ddee-4abd-982f-0d84f23b7926" (UID: "a04c85b4-ddee-4abd-982f-0d84f23b7926"). InnerVolumeSpecName "kube-api-access-j6jqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:17.406796 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.406776    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a04c85b4-ddee-4abd-982f-0d84f23b7926-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a04c85b4-ddee-4abd-982f-0d84f23b7926" (UID: "a04c85b4-ddee-4abd-982f-0d84f23b7926"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:17.406854 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.406797    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a04c85b4-ddee-4abd-982f-0d84f23b7926-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a04c85b4-ddee-4abd-982f-0d84f23b7926" (UID: "a04c85b4-ddee-4abd-982f-0d84f23b7926"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:17.505576 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.505538    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a04c85b4-ddee-4abd-982f-0d84f23b7926-oauth-serving-cert\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:42:17.505576 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.505567    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a04c85b4-ddee-4abd-982f-0d84f23b7926-console-oauth-config\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:42:17.505576 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.505576    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a04c85b4-ddee-4abd-982f-0d84f23b7926-console-serving-cert\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:42:17.505576 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.505584    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6jqj\" (UniqueName: \"kubernetes.io/projected/a04c85b4-ddee-4abd-982f-0d84f23b7926-kube-api-access-j6jqj\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:42:17.505828 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.505594    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a04c85b4-ddee-4abd-982f-0d84f23b7926-service-ca\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:42:17.961308 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.961268    2565 generic.go:356] "Generic (PLEG): container finished" podID="a04c85b4-ddee-4abd-982f-0d84f23b7926" containerID="8a682a1c53d34c37d531a275aac7c09f397896ba1b62a53f32e87fe1d0104bea" exitCode=0
Sep 16 18:42:17.961520 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.961320    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5754bd5445-lvdtm"
Sep 16 18:42:17.961520 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.961337    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5754bd5445-lvdtm" event={"ID":"a04c85b4-ddee-4abd-982f-0d84f23b7926","Type":"ContainerDied","Data":"8a682a1c53d34c37d531a275aac7c09f397896ba1b62a53f32e87fe1d0104bea"}
Sep 16 18:42:17.961520 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.961445    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5754bd5445-lvdtm" event={"ID":"a04c85b4-ddee-4abd-982f-0d84f23b7926","Type":"ContainerDied","Data":"9fd689d5a0d04ecb4e5f0003b188eef3e1cfa0376e10c5e084e23440efbd475c"}
Sep 16 18:42:17.961520 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.961474    2565 scope.go:117] "RemoveContainer" containerID="8a682a1c53d34c37d531a275aac7c09f397896ba1b62a53f32e87fe1d0104bea"
Sep 16 18:42:17.970778 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.970751    2565 scope.go:117] "RemoveContainer" containerID="8a682a1c53d34c37d531a275aac7c09f397896ba1b62a53f32e87fe1d0104bea"
Sep 16 18:42:17.971097 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:42:17.971078    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a682a1c53d34c37d531a275aac7c09f397896ba1b62a53f32e87fe1d0104bea\": container with ID starting with 8a682a1c53d34c37d531a275aac7c09f397896ba1b62a53f32e87fe1d0104bea not found: ID does not exist" containerID="8a682a1c53d34c37d531a275aac7c09f397896ba1b62a53f32e87fe1d0104bea"
Sep 16 18:42:17.971167 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.971105    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a682a1c53d34c37d531a275aac7c09f397896ba1b62a53f32e87fe1d0104bea"} err="failed to get container status \"8a682a1c53d34c37d531a275aac7c09f397896ba1b62a53f32e87fe1d0104bea\": rpc error: code = NotFound desc = could not find container \"8a682a1c53d34c37d531a275aac7c09f397896ba1b62a53f32e87fe1d0104bea\": container with ID starting with 8a682a1c53d34c37d531a275aac7c09f397896ba1b62a53f32e87fe1d0104bea not found: ID does not exist"
Sep 16 18:42:17.990726 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.990693    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5754bd5445-lvdtm"]
Sep 16 18:42:17.994992 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:17.994965    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5754bd5445-lvdtm"]
Sep 16 18:42:18.206408 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:18.206366    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a04c85b4-ddee-4abd-982f-0d84f23b7926" path="/var/lib/kubelet/pods/a04c85b4-ddee-4abd-982f-0d84f23b7926/volumes"
Sep 16 18:42:37.741037 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.740997    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:42:37.741484 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.741331    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a04c85b4-ddee-4abd-982f-0d84f23b7926" containerName="console"
Sep 16 18:42:37.741484 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.741357    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a04c85b4-ddee-4abd-982f-0d84f23b7926" containerName="console"
Sep 16 18:42:37.741484 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.741417    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a04c85b4-ddee-4abd-982f-0d84f23b7926" containerName="console"
Sep 16 18:42:37.746317 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.746294    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.750312 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.750286    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:42:37.750832 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.750814    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:42:37.751023 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.751008    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:42:37.751171 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.751151    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:42:37.751298 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.751280    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:42:37.751444 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.751422    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:42:37.751444 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.751440    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qljqg\""
Sep 16 18:42:37.751568 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.751422    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:42:37.751568 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.751544    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:42:37.761216 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.761190    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:42:37.766326 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.766296    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:42:37.881411 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.881332    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77b07526-8e5e-4d6d-915f-06ff138a9e8a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.881603 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.881430    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/77b07526-8e5e-4d6d-915f-06ff138a9e8a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.881603 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.881454    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.881603 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.881514    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.881603 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.881537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.881603 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.881561    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzrv5\" (UniqueName: \"kubernetes.io/projected/77b07526-8e5e-4d6d-915f-06ff138a9e8a-kube-api-access-xzrv5\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.881603 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.881596    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/77b07526-8e5e-4d6d-915f-06ff138a9e8a-config-out\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.881848 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.881614    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-web-config\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.881848 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.881630    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77b07526-8e5e-4d6d-915f-06ff138a9e8a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.881848 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.881648    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-config-volume\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.881848 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.881667    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.881848 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.881742    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.881848 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.881785    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/77b07526-8e5e-4d6d-915f-06ff138a9e8a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.982847 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.982807    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/77b07526-8e5e-4d6d-915f-06ff138a9e8a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.982847 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.982854    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.983072 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.982981    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.983072 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.983027    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.983072 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.983062    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzrv5\" (UniqueName: \"kubernetes.io/projected/77b07526-8e5e-4d6d-915f-06ff138a9e8a-kube-api-access-xzrv5\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.983338 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.983296    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/77b07526-8e5e-4d6d-915f-06ff138a9e8a-config-out\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.983524 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.983370    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-web-config\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.983524 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.983400    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77b07526-8e5e-4d6d-915f-06ff138a9e8a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.983524 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.983440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-config-volume\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.983524 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.983468    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.983524 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.983519    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.983793 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.983556    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/77b07526-8e5e-4d6d-915f-06ff138a9e8a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.983793 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.983638    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77b07526-8e5e-4d6d-915f-06ff138a9e8a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.985103 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.985071    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77b07526-8e5e-4d6d-915f-06ff138a9e8a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.985382 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.985329    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77b07526-8e5e-4d6d-915f-06ff138a9e8a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.985681 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.985659    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/77b07526-8e5e-4d6d-915f-06ff138a9e8a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.986184 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.986161    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/77b07526-8e5e-4d6d-915f-06ff138a9e8a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.986421 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.986395    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.986531 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.986473    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.986684 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.986655    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/77b07526-8e5e-4d6d-915f-06ff138a9e8a-config-out\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.986955 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.986934    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.987634 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.987604    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-web-config\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.987782 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.987762    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.987991 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.987969    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.988284 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.988236    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/77b07526-8e5e-4d6d-915f-06ff138a9e8a-config-volume\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:37.992890 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:37.992833    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzrv5\" (UniqueName: \"kubernetes.io/projected/77b07526-8e5e-4d6d-915f-06ff138a9e8a-kube-api-access-xzrv5\") pod \"alertmanager-main-0\" (UID: \"77b07526-8e5e-4d6d-915f-06ff138a9e8a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:38.056214 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:38.056173    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:42:38.351491 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:38.351464    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:42:38.353897 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:42:38.353864    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b07526_8e5e_4d6d_915f_06ff138a9e8a.slice/crio-7ed29e44d9a5b69f170580fa46eff9fc53a7c325a7ba202815a35502c0e8164d WatchSource:0}: Error finding container 7ed29e44d9a5b69f170580fa46eff9fc53a7c325a7ba202815a35502c0e8164d: Status 404 returned error can't find the container with id 7ed29e44d9a5b69f170580fa46eff9fc53a7c325a7ba202815a35502c0e8164d
Sep 16 18:42:39.053556 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:39.053506    2565 generic.go:356] "Generic (PLEG): container finished" podID="77b07526-8e5e-4d6d-915f-06ff138a9e8a" containerID="f3ce2dd7561da2c17ebf3a7235cc5b31a34f009ac6a983e534621e47870bc271" exitCode=0
Sep 16 18:42:39.053979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:39.053567    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"77b07526-8e5e-4d6d-915f-06ff138a9e8a","Type":"ContainerDied","Data":"f3ce2dd7561da2c17ebf3a7235cc5b31a34f009ac6a983e534621e47870bc271"}
Sep 16 18:42:39.053979 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:39.053638    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"77b07526-8e5e-4d6d-915f-06ff138a9e8a","Type":"ContainerStarted","Data":"7ed29e44d9a5b69f170580fa46eff9fc53a7c325a7ba202815a35502c0e8164d"}
Sep 16 18:42:41.066402 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:41.066361    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"77b07526-8e5e-4d6d-915f-06ff138a9e8a","Type":"ContainerStarted","Data":"a0647f8a6aec47d15529196096b31230a1a3bfa8fe573496683846ba8008d9af"}
Sep 16 18:42:41.066887 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:41.066457    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"77b07526-8e5e-4d6d-915f-06ff138a9e8a","Type":"ContainerStarted","Data":"5ffa2e69bce457d2acf24de4f3dd37240877fdf76557ae97fbfe771d636b7451"}
Sep 16 18:42:41.066887 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:41.066504    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"77b07526-8e5e-4d6d-915f-06ff138a9e8a","Type":"ContainerStarted","Data":"72c598cd43c02bc3e9a1008d9f2283de553100d5d0223fe39b5c5914b198d2dc"}
Sep 16 18:42:41.066887 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:41.066544    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"77b07526-8e5e-4d6d-915f-06ff138a9e8a","Type":"ContainerStarted","Data":"8b35c5a1b374c53cd651d851b8460853a402189ea4fcae19b0cabd459e2ac579"}
Sep 16 18:42:41.066887 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:41.066585    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"77b07526-8e5e-4d6d-915f-06ff138a9e8a","Type":"ContainerStarted","Data":"0c9a3318a70e1255a70013a8340cb78008c93cb8cd519a630d6086d6e459488c"}
Sep 16 18:42:42.073649 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:42.073604    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"77b07526-8e5e-4d6d-915f-06ff138a9e8a","Type":"ContainerStarted","Data":"cc8a6d42ab5ace42803baaa7a375451fb53c97ad5c3f214bbf10ae8c87eee786"}
Sep 16 18:42:42.109823 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:42.109760    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.111550759 podStartE2EDuration="5.109738994s" totalImagesPullingTime="1.978702764s" totalInitContainerRuntime="19.485471ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:42.109308873 +0000 UTC m=+200.468826881" watchObservedRunningTime="2026-09-16 18:42:42.109738994 +0000 UTC m=+200.469256990"
Sep 16 18:42:56.717434 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.717392    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fbb95445f-hsgwv"]
Sep 16 18:42:56.720930 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.720898    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.740305 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.740268    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fbb95445f-hsgwv"]
Sep 16 18:42:56.747740 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.747710    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-console-config\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.747740 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.747741    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-service-ca\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.747974 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.747759    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-oauth-serving-cert\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.747974 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.747777    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ca8f66d-25e2-4a13-9476-f731e526e11b-console-oauth-config\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.747974 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.747884    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ca8f66d-25e2-4a13-9476-f731e526e11b-console-serving-cert\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.747974 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.747920    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-trusted-ca-bundle\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.748116 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.747981    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbk4v\" (UniqueName: \"kubernetes.io/projected/7ca8f66d-25e2-4a13-9476-f731e526e11b-kube-api-access-fbk4v\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.848986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.848948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbk4v\" (UniqueName: \"kubernetes.io/projected/7ca8f66d-25e2-4a13-9476-f731e526e11b-kube-api-access-fbk4v\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.849198 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.849011    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-console-config\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.849198 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.849029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-service-ca\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.849198 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.849048    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-oauth-serving-cert\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.849198 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.849073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ca8f66d-25e2-4a13-9476-f731e526e11b-console-oauth-config\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.849198 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.849127    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ca8f66d-25e2-4a13-9476-f731e526e11b-console-serving-cert\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.849198 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.849157    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-trusted-ca-bundle\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.849916 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.849888    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-oauth-serving-cert\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.850025 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.849888    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-service-ca\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.850025 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.849994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-console-config\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.850111 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.850095    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-trusted-ca-bundle\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.852219 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.852194    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ca8f66d-25e2-4a13-9476-f731e526e11b-console-oauth-config\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.852326 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.852311    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ca8f66d-25e2-4a13-9476-f731e526e11b-console-serving-cert\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:56.858984 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:56.858959    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbk4v\" (UniqueName: \"kubernetes.io/projected/7ca8f66d-25e2-4a13-9476-f731e526e11b-kube-api-access-fbk4v\") pod \"console-6fbb95445f-hsgwv\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") " pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:57.029811 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:57.029767    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:42:57.311584 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:57.311504    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fbb95445f-hsgwv"]
Sep 16 18:42:57.315463 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:42:57.315434    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca8f66d_25e2_4a13_9476_f731e526e11b.slice/crio-e5c23e1070823eb5e4a1a335d347eb4db27ed54462315aed44864d873988a5ec WatchSource:0}: Error finding container e5c23e1070823eb5e4a1a335d347eb4db27ed54462315aed44864d873988a5ec: Status 404 returned error can't find the container with id e5c23e1070823eb5e4a1a335d347eb4db27ed54462315aed44864d873988a5ec
Sep 16 18:42:58.139503 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:58.139459    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbb95445f-hsgwv" event={"ID":"7ca8f66d-25e2-4a13-9476-f731e526e11b","Type":"ContainerStarted","Data":"4032062f14ae2f245fb43cb088f80251defa383bba56ca1caa6966225453a031"}
Sep 16 18:42:58.140002 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:58.139525    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbb95445f-hsgwv" event={"ID":"7ca8f66d-25e2-4a13-9476-f731e526e11b","Type":"ContainerStarted","Data":"e5c23e1070823eb5e4a1a335d347eb4db27ed54462315aed44864d873988a5ec"}
Sep 16 18:42:58.167042 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:42:58.166989    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fbb95445f-hsgwv" podStartSLOduration=2.166972493 podStartE2EDuration="2.166972493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:58.166611019 +0000 UTC m=+216.526129015" watchObservedRunningTime="2026-09-16 18:42:58.166972493 +0000 UTC m=+216.526490561"
Sep 16 18:43:07.030448 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:07.030404    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:43:07.030964 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:07.030473    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:43:07.035632 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:07.035599    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:43:07.179038 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:07.179004    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:43:07.246685 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:07.246643    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57db5fbc7f-pv6f7"]
Sep 16 18:43:32.266379 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.266285    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57db5fbc7f-pv6f7" podUID="9547c8de-0189-4a45-b201-686ed43dfa29" containerName="console" containerID="cri-o://2e0bb0c25c691c96158c0f1f6345b5c270058d51f40fb86ae4e2dd05445bdcbe" gracePeriod=15
Sep 16 18:43:32.520008 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.519934    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:43:32.657251 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.657208    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-service-ca\") pod \"9547c8de-0189-4a45-b201-686ed43dfa29\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") "
Sep 16 18:43:32.657447 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.657297    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-trusted-ca-bundle\") pod \"9547c8de-0189-4a45-b201-686ed43dfa29\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") "
Sep 16 18:43:32.657447 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.657322    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-oauth-serving-cert\") pod \"9547c8de-0189-4a45-b201-686ed43dfa29\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") "
Sep 16 18:43:32.657447 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.657373    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-console-config\") pod \"9547c8de-0189-4a45-b201-686ed43dfa29\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") "
Sep 16 18:43:32.657447 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.657399    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltcz7\" (UniqueName: \"kubernetes.io/projected/9547c8de-0189-4a45-b201-686ed43dfa29-kube-api-access-ltcz7\") pod \"9547c8de-0189-4a45-b201-686ed43dfa29\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") "
Sep 16 18:43:32.657447 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.657432    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9547c8de-0189-4a45-b201-686ed43dfa29-console-serving-cert\") pod \"9547c8de-0189-4a45-b201-686ed43dfa29\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") "
Sep 16 18:43:32.657694 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.657461    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9547c8de-0189-4a45-b201-686ed43dfa29-console-oauth-config\") pod \"9547c8de-0189-4a45-b201-686ed43dfa29\" (UID: \"9547c8de-0189-4a45-b201-686ed43dfa29\") "
Sep 16 18:43:32.657816 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.657790    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-service-ca" (OuterVolumeSpecName: "service-ca") pod "9547c8de-0189-4a45-b201-686ed43dfa29" (UID: "9547c8de-0189-4a45-b201-686ed43dfa29"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:32.657878 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.657806    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9547c8de-0189-4a45-b201-686ed43dfa29" (UID: "9547c8de-0189-4a45-b201-686ed43dfa29"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:32.657935 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.657907    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9547c8de-0189-4a45-b201-686ed43dfa29" (UID: "9547c8de-0189-4a45-b201-686ed43dfa29"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:32.657983 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.657932    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-console-config" (OuterVolumeSpecName: "console-config") pod "9547c8de-0189-4a45-b201-686ed43dfa29" (UID: "9547c8de-0189-4a45-b201-686ed43dfa29"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:32.659768 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.659743    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9547c8de-0189-4a45-b201-686ed43dfa29-kube-api-access-ltcz7" (OuterVolumeSpecName: "kube-api-access-ltcz7") pod "9547c8de-0189-4a45-b201-686ed43dfa29" (UID: "9547c8de-0189-4a45-b201-686ed43dfa29"). InnerVolumeSpecName "kube-api-access-ltcz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:32.659839 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.659769    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9547c8de-0189-4a45-b201-686ed43dfa29-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9547c8de-0189-4a45-b201-686ed43dfa29" (UID: "9547c8de-0189-4a45-b201-686ed43dfa29"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:32.659839 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.659790    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9547c8de-0189-4a45-b201-686ed43dfa29-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9547c8de-0189-4a45-b201-686ed43dfa29" (UID: "9547c8de-0189-4a45-b201-686ed43dfa29"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:32.759074 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.759031    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-service-ca\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:43:32.759074 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.759065    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-trusted-ca-bundle\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:43:32.759074 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.759075    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-oauth-serving-cert\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:43:32.759074 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.759086    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9547c8de-0189-4a45-b201-686ed43dfa29-console-config\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:43:32.759326 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.759095    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltcz7\" (UniqueName: \"kubernetes.io/projected/9547c8de-0189-4a45-b201-686ed43dfa29-kube-api-access-ltcz7\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:43:32.759326 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.759103    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9547c8de-0189-4a45-b201-686ed43dfa29-console-serving-cert\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:43:32.759326 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:32.759112    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9547c8de-0189-4a45-b201-686ed43dfa29-console-oauth-config\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:43:33.283517 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:33.283477    2565 generic.go:356] "Generic (PLEG): container finished" podID="9547c8de-0189-4a45-b201-686ed43dfa29" containerID="2e0bb0c25c691c96158c0f1f6345b5c270058d51f40fb86ae4e2dd05445bdcbe" exitCode=0
Sep 16 18:43:33.283961 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:33.283523    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57db5fbc7f-pv6f7"
Sep 16 18:43:33.283961 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:33.283569    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57db5fbc7f-pv6f7" event={"ID":"9547c8de-0189-4a45-b201-686ed43dfa29","Type":"ContainerDied","Data":"2e0bb0c25c691c96158c0f1f6345b5c270058d51f40fb86ae4e2dd05445bdcbe"}
Sep 16 18:43:33.283961 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:33.283620    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57db5fbc7f-pv6f7" event={"ID":"9547c8de-0189-4a45-b201-686ed43dfa29","Type":"ContainerDied","Data":"ea4204c6d4fd0cd158a952a4e9c96b6685ed69ba4ca66be59547430d73aa7e4e"}
Sep 16 18:43:33.283961 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:33.283640    2565 scope.go:117] "RemoveContainer" containerID="2e0bb0c25c691c96158c0f1f6345b5c270058d51f40fb86ae4e2dd05445bdcbe"
Sep 16 18:43:33.292996 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:33.292978    2565 scope.go:117] "RemoveContainer" containerID="2e0bb0c25c691c96158c0f1f6345b5c270058d51f40fb86ae4e2dd05445bdcbe"
Sep 16 18:43:33.293320 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:43:33.293302    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0bb0c25c691c96158c0f1f6345b5c270058d51f40fb86ae4e2dd05445bdcbe\": container with ID starting with 2e0bb0c25c691c96158c0f1f6345b5c270058d51f40fb86ae4e2dd05445bdcbe not found: ID does not exist" containerID="2e0bb0c25c691c96158c0f1f6345b5c270058d51f40fb86ae4e2dd05445bdcbe"
Sep 16 18:43:33.293408 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:33.293332    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0bb0c25c691c96158c0f1f6345b5c270058d51f40fb86ae4e2dd05445bdcbe"} err="failed to get container status \"2e0bb0c25c691c96158c0f1f6345b5c270058d51f40fb86ae4e2dd05445bdcbe\": rpc error: code = NotFound desc = could not find container \"2e0bb0c25c691c96158c0f1f6345b5c270058d51f40fb86ae4e2dd05445bdcbe\": container with ID starting with 2e0bb0c25c691c96158c0f1f6345b5c270058d51f40fb86ae4e2dd05445bdcbe not found: ID does not exist"
Sep 16 18:43:33.306258 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:33.306219    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57db5fbc7f-pv6f7"]
Sep 16 18:43:33.310887 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:33.310860    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57db5fbc7f-pv6f7"]
Sep 16 18:43:34.206640 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:34.206605    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9547c8de-0189-4a45-b201-686ed43dfa29" path="/var/lib/kubelet/pods/9547c8de-0189-4a45-b201-686ed43dfa29/volumes"
Sep 16 18:43:51.457783 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:51.457698    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm"]
Sep 16 18:43:51.458137 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:51.458036    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9547c8de-0189-4a45-b201-686ed43dfa29" containerName="console"
Sep 16 18:43:51.458137 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:51.458048    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9547c8de-0189-4a45-b201-686ed43dfa29" containerName="console"
Sep 16 18:43:51.458137 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:51.458109    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9547c8de-0189-4a45-b201-686ed43dfa29" containerName="console"
Sep 16 18:43:51.461171 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:51.461153    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm"
Sep 16 18:43:51.464113 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:51.464087    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-55b4r\""
Sep 16 18:43:51.464113 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:51.464101    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:43:51.464892 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:51.464874    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:43:51.474272 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:51.474243    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm"]
Sep 16 18:43:51.611880 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:51.611840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm\" (UID: \"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm"
Sep 16 18:43:51.612046 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:51.611917    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm\" (UID: \"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm"
Sep 16 18:43:51.612046 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:51.611957    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs4fd\" (UniqueName: \"kubernetes.io/projected/3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77-kube-api-access-hs4fd\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm\" (UID: \"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm"
Sep 16 18:43:51.713275 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:51.713187    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4fd\" (UniqueName: \"kubernetes.io/projected/3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77-kube-api-access-hs4fd\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm\" (UID: \"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm"
Sep 16 18:43:51.713275 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:51.713258    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm\" (UID: \"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm"
Sep 16 18:43:51.713473 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:51.713317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm\" (UID: \"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm"
Sep 16 18:43:51.713666 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:51.713648    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm\" (UID: \"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm"
Sep 16 18:43:51.713726 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:51.713708    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm\" (UID: \"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm"
Sep 16 18:43:51.724383 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:51.724333    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs4fd\" (UniqueName: \"kubernetes.io/projected/3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77-kube-api-access-hs4fd\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm\" (UID: \"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm"
Sep 16 18:43:51.769891 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:51.769852    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm"
Sep 16 18:43:52.043933 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:52.043907    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm"]
Sep 16 18:43:52.046431 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:43:52.046403    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c35cacc_5f0c_42f2_b3b2_d0fb108c1d77.slice/crio-e83aef689188f4159f6c9a3d51b3b0feea4ea6bf8a25315aebad7f5343f2f5ab WatchSource:0}: Error finding container e83aef689188f4159f6c9a3d51b3b0feea4ea6bf8a25315aebad7f5343f2f5ab: Status 404 returned error can't find the container with id e83aef689188f4159f6c9a3d51b3b0feea4ea6bf8a25315aebad7f5343f2f5ab
Sep 16 18:43:52.359360 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:52.359255    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm" event={"ID":"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77","Type":"ContainerStarted","Data":"e83aef689188f4159f6c9a3d51b3b0feea4ea6bf8a25315aebad7f5343f2f5ab"}
Sep 16 18:43:59.398718 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:59.398678    2565 generic.go:356] "Generic (PLEG): container finished" podID="3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77" containerID="7a19cdad0ef6d3ab2a0a1dd4ae431fd6d7ccdda5808314d1b032a843fc9c6a7c" exitCode=0
Sep 16 18:43:59.399088 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:43:59.398729    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm" event={"ID":"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77","Type":"ContainerDied","Data":"7a19cdad0ef6d3ab2a0a1dd4ae431fd6d7ccdda5808314d1b032a843fc9c6a7c"}
Sep 16 18:44:04.427701 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:04.427607    2565 generic.go:356] "Generic (PLEG): container finished" podID="3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77" containerID="73a9be249b8430372229315f73e11eae9274aa7f018631150b49a9188a3447b8" exitCode=0
Sep 16 18:44:04.427701 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:04.427658    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm" event={"ID":"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77","Type":"ContainerDied","Data":"73a9be249b8430372229315f73e11eae9274aa7f018631150b49a9188a3447b8"}
Sep 16 18:44:11.462096 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:11.462059    2565 generic.go:356] "Generic (PLEG): container finished" podID="3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77" containerID="abd070e7e4964761a0e80b71f96565229e1910863117647495cfd52cc4bceb7d" exitCode=0
Sep 16 18:44:11.462512 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:11.462112    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm" event={"ID":"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77","Type":"ContainerDied","Data":"abd070e7e4964761a0e80b71f96565229e1910863117647495cfd52cc4bceb7d"}
Sep 16 18:44:12.590183 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:12.590150    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm"
Sep 16 18:44:12.677105 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:12.677055    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77-bundle\") pod \"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77\" (UID: \"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77\") "
Sep 16 18:44:12.677105 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:12.677104    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs4fd\" (UniqueName: \"kubernetes.io/projected/3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77-kube-api-access-hs4fd\") pod \"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77\" (UID: \"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77\") "
Sep 16 18:44:12.677392 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:12.677146    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77-util\") pod \"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77\" (UID: \"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77\") "
Sep 16 18:44:12.677760 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:12.677730    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77-bundle" (OuterVolumeSpecName: "bundle") pod "3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77" (UID: "3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:12.679418 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:12.679393    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77-kube-api-access-hs4fd" (OuterVolumeSpecName: "kube-api-access-hs4fd") pod "3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77" (UID: "3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77"). InnerVolumeSpecName "kube-api-access-hs4fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:12.682269 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:12.682241    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77-util" (OuterVolumeSpecName: "util") pod "3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77" (UID: "3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:12.778011 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:12.777970    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77-bundle\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:44:12.778011 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:12.778008    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs4fd\" (UniqueName: \"kubernetes.io/projected/3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77-kube-api-access-hs4fd\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:44:12.778222 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:12.778020    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77-util\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:44:13.472061 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:13.472031    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm"
Sep 16 18:44:13.472234 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:13.472032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apdfvm" event={"ID":"3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77","Type":"ContainerDied","Data":"e83aef689188f4159f6c9a3d51b3b0feea4ea6bf8a25315aebad7f5343f2f5ab"}
Sep 16 18:44:13.472234 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:13.472153    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e83aef689188f4159f6c9a3d51b3b0feea4ea6bf8a25315aebad7f5343f2f5ab"
Sep 16 18:44:19.939714 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:19.939672    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-m48th"]
Sep 16 18:44:19.940107 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:19.939980    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77" containerName="pull"
Sep 16 18:44:19.940107 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:19.939991    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77" containerName="pull"
Sep 16 18:44:19.940107 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:19.940006    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77" containerName="extract"
Sep 16 18:44:19.940107 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:19.940012    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77" containerName="extract"
Sep 16 18:44:19.940107 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:19.940023    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77" containerName="util"
Sep 16 18:44:19.940107 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:19.940028    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77" containerName="util"
Sep 16 18:44:19.940107 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:19.940095    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c35cacc-5f0c-42f2-b3b2-d0fb108c1d77" containerName="extract"
Sep 16 18:44:19.943020 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:19.943002    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-m48th"
Sep 16 18:44:19.946068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:19.946034    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 18:44:19.946202 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:19.946136    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-h2hjv\""
Sep 16 18:44:19.946202 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:19.946136    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 18:44:19.946338 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:19.946303    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 18:44:19.951869 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:19.951840    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-m48th"]
Sep 16 18:44:20.037583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:20.037537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl9xt\" (UniqueName: \"kubernetes.io/projected/ad0e6ac4-f711-45c7-ba0a-cd895c57a48c-kube-api-access-kl9xt\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-m48th\" (UID: \"ad0e6ac4-f711-45c7-ba0a-cd895c57a48c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-m48th"
Sep 16 18:44:20.037766 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:20.037606    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ad0e6ac4-f711-45c7-ba0a-cd895c57a48c-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-m48th\" (UID: \"ad0e6ac4-f711-45c7-ba0a-cd895c57a48c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-m48th"
Sep 16 18:44:20.138538 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:20.138475    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl9xt\" (UniqueName: \"kubernetes.io/projected/ad0e6ac4-f711-45c7-ba0a-cd895c57a48c-kube-api-access-kl9xt\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-m48th\" (UID: \"ad0e6ac4-f711-45c7-ba0a-cd895c57a48c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-m48th"
Sep 16 18:44:20.138770 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:20.138582    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ad0e6ac4-f711-45c7-ba0a-cd895c57a48c-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-m48th\" (UID: \"ad0e6ac4-f711-45c7-ba0a-cd895c57a48c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-m48th"
Sep 16 18:44:20.141023 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:20.140995    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ad0e6ac4-f711-45c7-ba0a-cd895c57a48c-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-m48th\" (UID: \"ad0e6ac4-f711-45c7-ba0a-cd895c57a48c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-m48th"
Sep 16 18:44:20.164082 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:20.164043    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl9xt\" (UniqueName: \"kubernetes.io/projected/ad0e6ac4-f711-45c7-ba0a-cd895c57a48c-kube-api-access-kl9xt\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-m48th\" (UID: \"ad0e6ac4-f711-45c7-ba0a-cd895c57a48c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-m48th"
Sep 16 18:44:20.254451 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:20.254414    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-m48th"
Sep 16 18:44:20.538665 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:20.538635    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-m48th"]
Sep 16 18:44:20.541526 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:44:20.541478    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0e6ac4_f711_45c7_ba0a_cd895c57a48c.slice/crio-522fd3e99e92ac4aeb4d427f4236ebc2361a64c1c09c1ada91eda173b54fc800 WatchSource:0}: Error finding container 522fd3e99e92ac4aeb4d427f4236ebc2361a64c1c09c1ada91eda173b54fc800: Status 404 returned error can't find the container with id 522fd3e99e92ac4aeb4d427f4236ebc2361a64c1c09c1ada91eda173b54fc800
Sep 16 18:44:21.507654 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:21.507613    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-m48th" event={"ID":"ad0e6ac4-f711-45c7-ba0a-cd895c57a48c","Type":"ContainerStarted","Data":"522fd3e99e92ac4aeb4d427f4236ebc2361a64c1c09c1ada91eda173b54fc800"}
Sep 16 18:44:22.081867 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:22.081824    2565 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:44:35.570164 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:35.570122    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-m48th" event={"ID":"ad0e6ac4-f711-45c7-ba0a-cd895c57a48c","Type":"ContainerStarted","Data":"c22fa49382906d3e95efca2240b74b520bd6616244543b0f911f1fa879e56c90"}
Sep 16 18:44:35.570699 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:35.570327    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-m48th"
Sep 16 18:44:36.763241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:36.763175    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-m48th" podStartSLOduration=3.18619169 podStartE2EDuration="17.763154924s" totalImagesPullingTime="14.576963234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:35.595295822 +0000 UTC m=+313.954813830" watchObservedRunningTime="2026-09-16 18:44:36.763154924 +0000 UTC m=+315.122672927"
Sep 16 18:44:36.764696 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:36.764677    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r"]
Sep 16 18:44:36.772338 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:36.772316    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r"
Sep 16 18:44:36.779058 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:36.779023    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 18:44:36.779220 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:36.779023    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 16 18:44:36.779287 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:36.779117    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-7sl84\""
Sep 16 18:44:36.786171 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:36.786133    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r"]
Sep 16 18:44:36.891263 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:36.891215    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c81c31bc-57f2-45bc-a2b5-973ade998329-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-bpc8r\" (UID: \"c81c31bc-57f2-45bc-a2b5-973ade998329\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r"
Sep 16 18:44:36.891449 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:36.891283    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rwfl\" (UniqueName: \"kubernetes.io/projected/c81c31bc-57f2-45bc-a2b5-973ade998329-kube-api-access-2rwfl\") pod \"keda-metrics-apiserver-77857b7b98-bpc8r\" (UID: \"c81c31bc-57f2-45bc-a2b5-973ade998329\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r"
Sep 16 18:44:36.891449 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:36.891372    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c81c31bc-57f2-45bc-a2b5-973ade998329-certificates\") pod \"keda-metrics-apiserver-77857b7b98-bpc8r\" (UID: \"c81c31bc-57f2-45bc-a2b5-973ade998329\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r"
Sep 16 18:44:36.992375 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:36.992308    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rwfl\" (UniqueName: \"kubernetes.io/projected/c81c31bc-57f2-45bc-a2b5-973ade998329-kube-api-access-2rwfl\") pod \"keda-metrics-apiserver-77857b7b98-bpc8r\" (UID: \"c81c31bc-57f2-45bc-a2b5-973ade998329\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r"
Sep 16 18:44:36.992590 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:36.992402    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c81c31bc-57f2-45bc-a2b5-973ade998329-certificates\") pod \"keda-metrics-apiserver-77857b7b98-bpc8r\" (UID: \"c81c31bc-57f2-45bc-a2b5-973ade998329\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r"
Sep 16 18:44:36.992590 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:36.992450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c81c31bc-57f2-45bc-a2b5-973ade998329-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-bpc8r\" (UID: \"c81c31bc-57f2-45bc-a2b5-973ade998329\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r"
Sep 16 18:44:36.992590 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:44:36.992565    2565 secret.go:281] references non-existent secret key: tls.crt
Sep 16 18:44:36.992742 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:44:36.992593    2565 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 16 18:44:36.992742 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:44:36.992616    2565 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r: references non-existent secret key: tls.crt
Sep 16 18:44:36.992742 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:44:36.992695    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c81c31bc-57f2-45bc-a2b5-973ade998329-certificates podName:c81c31bc-57f2-45bc-a2b5-973ade998329 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:37.49267397 +0000 UTC m=+315.852191955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c81c31bc-57f2-45bc-a2b5-973ade998329-certificates") pod "keda-metrics-apiserver-77857b7b98-bpc8r" (UID: "c81c31bc-57f2-45bc-a2b5-973ade998329") : references non-existent secret key: tls.crt
Sep 16 18:44:36.992907 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:36.992813    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c81c31bc-57f2-45bc-a2b5-973ade998329-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-bpc8r\" (UID: \"c81c31bc-57f2-45bc-a2b5-973ade998329\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r"
Sep 16 18:44:37.005133 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:37.005107    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rwfl\" (UniqueName: \"kubernetes.io/projected/c81c31bc-57f2-45bc-a2b5-973ade998329-kube-api-access-2rwfl\") pod \"keda-metrics-apiserver-77857b7b98-bpc8r\" (UID: \"c81c31bc-57f2-45bc-a2b5-973ade998329\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r"
Sep 16 18:44:37.476919 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:37.476876    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-lhwzk"]
Sep 16 18:44:37.480799 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:37.480778    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-lhwzk"
Sep 16 18:44:37.483788 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:37.483768    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 16 18:44:37.496741 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:37.496705    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c81c31bc-57f2-45bc-a2b5-973ade998329-certificates\") pod \"keda-metrics-apiserver-77857b7b98-bpc8r\" (UID: \"c81c31bc-57f2-45bc-a2b5-973ade998329\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r"
Sep 16 18:44:37.499445 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:37.499418    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c81c31bc-57f2-45bc-a2b5-973ade998329-certificates\") pod \"keda-metrics-apiserver-77857b7b98-bpc8r\" (UID: \"c81c31bc-57f2-45bc-a2b5-973ade998329\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r"
Sep 16 18:44:37.517016 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:37.516976    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-lhwzk"]
Sep 16 18:44:37.597815 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:37.597777    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c147e966-1471-4d69-b482-95d405f51cab-certificates\") pod \"keda-admission-6c4d864b86-lhwzk\" (UID: \"c147e966-1471-4d69-b482-95d405f51cab\") " pod="openshift-keda/keda-admission-6c4d864b86-lhwzk"
Sep 16 18:44:37.598001 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:37.597834    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdv9m\" (UniqueName: \"kubernetes.io/projected/c147e966-1471-4d69-b482-95d405f51cab-kube-api-access-kdv9m\") pod \"keda-admission-6c4d864b86-lhwzk\" (UID: \"c147e966-1471-4d69-b482-95d405f51cab\") " pod="openshift-keda/keda-admission-6c4d864b86-lhwzk"
Sep 16 18:44:37.689881 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:37.689840    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r"
Sep 16 18:44:37.698752 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:37.698716    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c147e966-1471-4d69-b482-95d405f51cab-certificates\") pod \"keda-admission-6c4d864b86-lhwzk\" (UID: \"c147e966-1471-4d69-b482-95d405f51cab\") " pod="openshift-keda/keda-admission-6c4d864b86-lhwzk"
Sep 16 18:44:37.698899 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:37.698767    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdv9m\" (UniqueName: \"kubernetes.io/projected/c147e966-1471-4d69-b482-95d405f51cab-kube-api-access-kdv9m\") pod \"keda-admission-6c4d864b86-lhwzk\" (UID: \"c147e966-1471-4d69-b482-95d405f51cab\") " pod="openshift-keda/keda-admission-6c4d864b86-lhwzk"
Sep 16 18:44:37.701277 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:37.701250    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c147e966-1471-4d69-b482-95d405f51cab-certificates\") pod \"keda-admission-6c4d864b86-lhwzk\" (UID: \"c147e966-1471-4d69-b482-95d405f51cab\") " pod="openshift-keda/keda-admission-6c4d864b86-lhwzk"
Sep 16 18:44:37.718444 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:37.718414    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdv9m\" (UniqueName: \"kubernetes.io/projected/c147e966-1471-4d69-b482-95d405f51cab-kube-api-access-kdv9m\") pod \"keda-admission-6c4d864b86-lhwzk\" (UID: \"c147e966-1471-4d69-b482-95d405f51cab\") " pod="openshift-keda/keda-admission-6c4d864b86-lhwzk"
Sep 16 18:44:37.791939 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:37.791894    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-lhwzk"
Sep 16 18:44:37.966900 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:37.966864    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r"]
Sep 16 18:44:37.970544 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:44:37.970507    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81c31bc_57f2_45bc_a2b5_973ade998329.slice/crio-66e2612e5cf5ed7d96d88611df555821a9fa3aa6838637da60381ebb131a240c WatchSource:0}: Error finding container 66e2612e5cf5ed7d96d88611df555821a9fa3aa6838637da60381ebb131a240c: Status 404 returned error can't find the container with id 66e2612e5cf5ed7d96d88611df555821a9fa3aa6838637da60381ebb131a240c
Sep 16 18:44:37.971771 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:37.971748    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:44:38.067281 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:38.067048    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-lhwzk"]
Sep 16 18:44:38.070267 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:44:38.070239    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc147e966_1471_4d69_b482_95d405f51cab.slice/crio-e3e8c12ba6f4c654b8b6938843698d83573c6189c35a3c8a1722b1548bd48832 WatchSource:0}: Error finding container e3e8c12ba6f4c654b8b6938843698d83573c6189c35a3c8a1722b1548bd48832: Status 404 returned error can't find the container with id e3e8c12ba6f4c654b8b6938843698d83573c6189c35a3c8a1722b1548bd48832
Sep 16 18:44:38.584992 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:38.584952    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r" event={"ID":"c81c31bc-57f2-45bc-a2b5-973ade998329","Type":"ContainerStarted","Data":"66e2612e5cf5ed7d96d88611df555821a9fa3aa6838637da60381ebb131a240c"}
Sep 16 18:44:38.586097 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:38.586077    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-lhwzk" event={"ID":"c147e966-1471-4d69-b482-95d405f51cab","Type":"ContainerStarted","Data":"e3e8c12ba6f4c654b8b6938843698d83573c6189c35a3c8a1722b1548bd48832"}
Sep 16 18:44:40.599138 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:40.599102    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-lhwzk" event={"ID":"c147e966-1471-4d69-b482-95d405f51cab","Type":"ContainerStarted","Data":"6f27c16856b0733b69e29df97ecec3a051dfc0be09eef538e635a739618e11bf"}
Sep 16 18:44:40.599656 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:40.599308    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-lhwzk"
Sep 16 18:44:40.620247 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:40.620182    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-lhwzk" podStartSLOduration=1.7423307559999999 podStartE2EDuration="3.620165405s" totalImagesPullingTime="1.877834649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:40.618366897 +0000 UTC m=+318.977884893" watchObservedRunningTime="2026-09-16 18:44:40.620165405 +0000 UTC m=+318.979683399"
Sep 16 18:44:42.609943 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:42.609902    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r" event={"ID":"c81c31bc-57f2-45bc-a2b5-973ade998329","Type":"ContainerStarted","Data":"e74a88ad1f734fd8787d742990b9be00076916afdd4b0af6934225cf259832cc"}
Sep 16 18:44:42.610387 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:42.610014    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r"
Sep 16 18:44:42.627740 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:42.627682    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r" podStartSLOduration=2.914422451 podStartE2EDuration="6.627664892s" totalImagesPullingTime="3.713242441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:42.626890694 +0000 UTC m=+320.986408712" watchObservedRunningTime="2026-09-16 18:44:42.627664892 +0000 UTC m=+320.987182887"
Sep 16 18:44:53.619583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:53.619554    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-bpc8r"
Sep 16 18:44:56.577553 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:44:56.577517    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-m48th"
Sep 16 18:45:01.606170 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:01.606136    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-lhwzk"
Sep 16 18:45:27.512307 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:27.512222    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55"]
Sep 16 18:45:27.515712 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:27.515693    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55"
Sep 16 18:45:27.518454 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:27.518426    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-55b4r\""
Sep 16 18:45:27.518582 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:27.518456    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:45:27.519226 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:27.519209    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:45:27.525221 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:27.525192    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55"]
Sep 16 18:45:27.624066 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:27.624031    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk55c\" (UniqueName: \"kubernetes.io/projected/2fdb814a-5d94-4d29-b65c-b27a61152263-kube-api-access-dk55c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55\" (UID: \"2fdb814a-5d94-4d29-b65c-b27a61152263\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55"
Sep 16 18:45:27.624066 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:27.624082    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fdb814a-5d94-4d29-b65c-b27a61152263-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55\" (UID: \"2fdb814a-5d94-4d29-b65c-b27a61152263\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55"
Sep 16 18:45:27.624374 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:27.624139    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fdb814a-5d94-4d29-b65c-b27a61152263-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55\" (UID: \"2fdb814a-5d94-4d29-b65c-b27a61152263\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55"
Sep 16 18:45:27.725003 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:27.724956    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk55c\" (UniqueName: \"kubernetes.io/projected/2fdb814a-5d94-4d29-b65c-b27a61152263-kube-api-access-dk55c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55\" (UID: \"2fdb814a-5d94-4d29-b65c-b27a61152263\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55"
Sep 16 18:45:27.725176 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:27.725013    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fdb814a-5d94-4d29-b65c-b27a61152263-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55\" (UID: \"2fdb814a-5d94-4d29-b65c-b27a61152263\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55"
Sep 16 18:45:27.725176 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:27.725048    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fdb814a-5d94-4d29-b65c-b27a61152263-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55\" (UID: \"2fdb814a-5d94-4d29-b65c-b27a61152263\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55"
Sep 16 18:45:27.725489 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:27.725469    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fdb814a-5d94-4d29-b65c-b27a61152263-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55\" (UID: \"2fdb814a-5d94-4d29-b65c-b27a61152263\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55"
Sep 16 18:45:27.725538 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:27.725496    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fdb814a-5d94-4d29-b65c-b27a61152263-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55\" (UID: \"2fdb814a-5d94-4d29-b65c-b27a61152263\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55"
Sep 16 18:45:27.733489 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:27.733457    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk55c\" (UniqueName: \"kubernetes.io/projected/2fdb814a-5d94-4d29-b65c-b27a61152263-kube-api-access-dk55c\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55\" (UID: \"2fdb814a-5d94-4d29-b65c-b27a61152263\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55"
Sep 16 18:45:27.825798 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:27.825704    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55"
Sep 16 18:45:28.098690 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:28.098667    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55"]
Sep 16 18:45:28.100982 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:45:28.100953    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fdb814a_5d94_4d29_b65c_b27a61152263.slice/crio-43cab52d53a96b4f5a19899700b323011c1493c1a17ba593c4cf2eadf1c08d3d WatchSource:0}: Error finding container 43cab52d53a96b4f5a19899700b323011c1493c1a17ba593c4cf2eadf1c08d3d: Status 404 returned error can't find the container with id 43cab52d53a96b4f5a19899700b323011c1493c1a17ba593c4cf2eadf1c08d3d
Sep 16 18:45:28.822454 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:28.822415    2565 generic.go:356] "Generic (PLEG): container finished" podID="2fdb814a-5d94-4d29-b65c-b27a61152263" containerID="58e67a82ab19ee48bc254b8ec9e06e9816c3ae165344c52fc684eb2c22a5b399" exitCode=0
Sep 16 18:45:28.822840 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:28.822477    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55" event={"ID":"2fdb814a-5d94-4d29-b65c-b27a61152263","Type":"ContainerDied","Data":"58e67a82ab19ee48bc254b8ec9e06e9816c3ae165344c52fc684eb2c22a5b399"}
Sep 16 18:45:28.822840 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:28.822551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55" event={"ID":"2fdb814a-5d94-4d29-b65c-b27a61152263","Type":"ContainerStarted","Data":"43cab52d53a96b4f5a19899700b323011c1493c1a17ba593c4cf2eadf1c08d3d"}
Sep 16 18:45:31.839723 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:31.839683    2565 generic.go:356] "Generic (PLEG): container finished" podID="2fdb814a-5d94-4d29-b65c-b27a61152263" containerID="ae81318de93515665e557d786f1cc059f292317e76009b3f148766f2421aa252" exitCode=0
Sep 16 18:45:31.840132 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:31.839769    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55" event={"ID":"2fdb814a-5d94-4d29-b65c-b27a61152263","Type":"ContainerDied","Data":"ae81318de93515665e557d786f1cc059f292317e76009b3f148766f2421aa252"}
Sep 16 18:45:32.847068 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:32.847028    2565 generic.go:356] "Generic (PLEG): container finished" podID="2fdb814a-5d94-4d29-b65c-b27a61152263" containerID="76ebde3e4264e0cf662416108be596f67d58933445e8d8f91ddb3b30f4845ab8" exitCode=0
Sep 16 18:45:32.847555 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:32.847092    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55" event={"ID":"2fdb814a-5d94-4d29-b65c-b27a61152263","Type":"ContainerDied","Data":"76ebde3e4264e0cf662416108be596f67d58933445e8d8f91ddb3b30f4845ab8"}
Sep 16 18:45:33.980028 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:33.979997    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55"
Sep 16 18:45:34.084980 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:34.084945    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fdb814a-5d94-4d29-b65c-b27a61152263-util\") pod \"2fdb814a-5d94-4d29-b65c-b27a61152263\" (UID: \"2fdb814a-5d94-4d29-b65c-b27a61152263\") "
Sep 16 18:45:34.085148 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:34.085003    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk55c\" (UniqueName: \"kubernetes.io/projected/2fdb814a-5d94-4d29-b65c-b27a61152263-kube-api-access-dk55c\") pod \"2fdb814a-5d94-4d29-b65c-b27a61152263\" (UID: \"2fdb814a-5d94-4d29-b65c-b27a61152263\") "
Sep 16 18:45:34.085148 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:34.085034    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fdb814a-5d94-4d29-b65c-b27a61152263-bundle\") pod \"2fdb814a-5d94-4d29-b65c-b27a61152263\" (UID: \"2fdb814a-5d94-4d29-b65c-b27a61152263\") "
Sep 16 18:45:34.085851 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:34.085825    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fdb814a-5d94-4d29-b65c-b27a61152263-bundle" (OuterVolumeSpecName: "bundle") pod "2fdb814a-5d94-4d29-b65c-b27a61152263" (UID: "2fdb814a-5d94-4d29-b65c-b27a61152263"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:34.087265 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:34.087244    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fdb814a-5d94-4d29-b65c-b27a61152263-kube-api-access-dk55c" (OuterVolumeSpecName: "kube-api-access-dk55c") pod "2fdb814a-5d94-4d29-b65c-b27a61152263" (UID: "2fdb814a-5d94-4d29-b65c-b27a61152263"). InnerVolumeSpecName "kube-api-access-dk55c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:34.089543 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:34.089507    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fdb814a-5d94-4d29-b65c-b27a61152263-util" (OuterVolumeSpecName: "util") pod "2fdb814a-5d94-4d29-b65c-b27a61152263" (UID: "2fdb814a-5d94-4d29-b65c-b27a61152263"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:34.186550 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:34.186457    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk55c\" (UniqueName: \"kubernetes.io/projected/2fdb814a-5d94-4d29-b65c-b27a61152263-kube-api-access-dk55c\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:45:34.186550 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:34.186488    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fdb814a-5d94-4d29-b65c-b27a61152263-bundle\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:45:34.186550 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:34.186499    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fdb814a-5d94-4d29-b65c-b27a61152263-util\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:45:34.857773 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:34.857736    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55" event={"ID":"2fdb814a-5d94-4d29-b65c-b27a61152263","Type":"ContainerDied","Data":"43cab52d53a96b4f5a19899700b323011c1493c1a17ba593c4cf2eadf1c08d3d"}
Sep 16 18:45:34.857948 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:34.857790    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43cab52d53a96b4f5a19899700b323011c1493c1a17ba593c4cf2eadf1c08d3d"
Sep 16 18:45:34.857948 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:34.857753    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tbm55"
Sep 16 18:45:49.665237 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.665197    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj"]
Sep 16 18:45:49.665642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.665565    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fdb814a-5d94-4d29-b65c-b27a61152263" containerName="extract"
Sep 16 18:45:49.665642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.665583    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fdb814a-5d94-4d29-b65c-b27a61152263" containerName="extract"
Sep 16 18:45:49.665642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.665603    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fdb814a-5d94-4d29-b65c-b27a61152263" containerName="pull"
Sep 16 18:45:49.665642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.665612    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fdb814a-5d94-4d29-b65c-b27a61152263" containerName="pull"
Sep 16 18:45:49.665642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.665635    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fdb814a-5d94-4d29-b65c-b27a61152263" containerName="util"
Sep 16 18:45:49.665642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.665644    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fdb814a-5d94-4d29-b65c-b27a61152263" containerName="util"
Sep 16 18:45:49.665824 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.665735    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fdb814a-5d94-4d29-b65c-b27a61152263" containerName="extract"
Sep 16 18:45:49.672683 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.672661    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj"
Sep 16 18:45:49.675982 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.675950    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:45:49.676117 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.675977    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:45:49.676849 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.676826    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-55b4r\""
Sep 16 18:45:49.684430 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.684401    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj"]
Sep 16 18:45:49.827635 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.827589    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4572a53-c77c-4232-9d45-f74d534c98aa-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj\" (UID: \"c4572a53-c77c-4232-9d45-f74d534c98aa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj"
Sep 16 18:45:49.827840 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.827691    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4572a53-c77c-4232-9d45-f74d534c98aa-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj\" (UID: \"c4572a53-c77c-4232-9d45-f74d534c98aa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj"
Sep 16 18:45:49.827840 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.827721    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whtvj\" (UniqueName: \"kubernetes.io/projected/c4572a53-c77c-4232-9d45-f74d534c98aa-kube-api-access-whtvj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj\" (UID: \"c4572a53-c77c-4232-9d45-f74d534c98aa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj"
Sep 16 18:45:49.928643 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.928552    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4572a53-c77c-4232-9d45-f74d534c98aa-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj\" (UID: \"c4572a53-c77c-4232-9d45-f74d534c98aa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj"
Sep 16 18:45:49.928643 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.928632    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4572a53-c77c-4232-9d45-f74d534c98aa-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj\" (UID: \"c4572a53-c77c-4232-9d45-f74d534c98aa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj"
Sep 16 18:45:49.928882 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.928664    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whtvj\" (UniqueName: \"kubernetes.io/projected/c4572a53-c77c-4232-9d45-f74d534c98aa-kube-api-access-whtvj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj\" (UID: \"c4572a53-c77c-4232-9d45-f74d534c98aa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj"
Sep 16 18:45:49.929055 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.929027    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4572a53-c77c-4232-9d45-f74d534c98aa-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj\" (UID: \"c4572a53-c77c-4232-9d45-f74d534c98aa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj"
Sep 16 18:45:49.929123 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.929059    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4572a53-c77c-4232-9d45-f74d534c98aa-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj\" (UID: \"c4572a53-c77c-4232-9d45-f74d534c98aa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj"
Sep 16 18:45:49.951871 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.951837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whtvj\" (UniqueName: \"kubernetes.io/projected/c4572a53-c77c-4232-9d45-f74d534c98aa-kube-api-access-whtvj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj\" (UID: \"c4572a53-c77c-4232-9d45-f74d534c98aa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj"
Sep 16 18:45:49.982449 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:49.982415    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj"
Sep 16 18:45:50.267226 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:50.267196    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj"]
Sep 16 18:45:50.269500 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:45:50.269470    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4572a53_c77c_4232_9d45_f74d534c98aa.slice/crio-82234da06f9d26fb1f59e1a0728828ce1220d2bf02d64127289c93437ca76e24 WatchSource:0}: Error finding container 82234da06f9d26fb1f59e1a0728828ce1220d2bf02d64127289c93437ca76e24: Status 404 returned error can't find the container with id 82234da06f9d26fb1f59e1a0728828ce1220d2bf02d64127289c93437ca76e24
Sep 16 18:45:50.932431 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:50.932392    2565 generic.go:356] "Generic (PLEG): container finished" podID="c4572a53-c77c-4232-9d45-f74d534c98aa" containerID="a3b884d40642f911d8bd6f1fceb0c2e434548caee1aa7792d1e1108dd4a6cd8a" exitCode=0
Sep 16 18:45:50.932928 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:50.932485    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj" event={"ID":"c4572a53-c77c-4232-9d45-f74d534c98aa","Type":"ContainerDied","Data":"a3b884d40642f911d8bd6f1fceb0c2e434548caee1aa7792d1e1108dd4a6cd8a"}
Sep 16 18:45:50.932928 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:50.932533    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj" event={"ID":"c4572a53-c77c-4232-9d45-f74d534c98aa","Type":"ContainerStarted","Data":"82234da06f9d26fb1f59e1a0728828ce1220d2bf02d64127289c93437ca76e24"}
Sep 16 18:45:53.949241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:53.949199    2565 generic.go:356] "Generic (PLEG): container finished" podID="c4572a53-c77c-4232-9d45-f74d534c98aa" containerID="d7eccb105c8fc2b51b35f2f66c32165f119465eb056f15d9e7ef2ff6050dadf0" exitCode=0
Sep 16 18:45:53.949796 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:53.949298    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj" event={"ID":"c4572a53-c77c-4232-9d45-f74d534c98aa","Type":"ContainerDied","Data":"d7eccb105c8fc2b51b35f2f66c32165f119465eb056f15d9e7ef2ff6050dadf0"}
Sep 16 18:45:54.956403 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:54.956368    2565 generic.go:356] "Generic (PLEG): container finished" podID="c4572a53-c77c-4232-9d45-f74d534c98aa" containerID="1015695f823edd4a32c1672604cfc52752fc759173e221a4dfafb2c792c2faf7" exitCode=0
Sep 16 18:45:54.956806 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:54.956430    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj" event={"ID":"c4572a53-c77c-4232-9d45-f74d534c98aa","Type":"ContainerDied","Data":"1015695f823edd4a32c1672604cfc52752fc759173e221a4dfafb2c792c2faf7"}
Sep 16 18:45:56.087953 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:56.087900    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj"
Sep 16 18:45:56.286053 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:56.286022    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4572a53-c77c-4232-9d45-f74d534c98aa-util\") pod \"c4572a53-c77c-4232-9d45-f74d534c98aa\" (UID: \"c4572a53-c77c-4232-9d45-f74d534c98aa\") "
Sep 16 18:45:56.286206 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:56.286098    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4572a53-c77c-4232-9d45-f74d534c98aa-bundle\") pod \"c4572a53-c77c-4232-9d45-f74d534c98aa\" (UID: \"c4572a53-c77c-4232-9d45-f74d534c98aa\") "
Sep 16 18:45:56.286206 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:56.286158    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whtvj\" (UniqueName: \"kubernetes.io/projected/c4572a53-c77c-4232-9d45-f74d534c98aa-kube-api-access-whtvj\") pod \"c4572a53-c77c-4232-9d45-f74d534c98aa\" (UID: \"c4572a53-c77c-4232-9d45-f74d534c98aa\") "
Sep 16 18:45:56.286522 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:56.286499    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4572a53-c77c-4232-9d45-f74d534c98aa-bundle" (OuterVolumeSpecName: "bundle") pod "c4572a53-c77c-4232-9d45-f74d534c98aa" (UID: "c4572a53-c77c-4232-9d45-f74d534c98aa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:56.288387 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:56.288335    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4572a53-c77c-4232-9d45-f74d534c98aa-kube-api-access-whtvj" (OuterVolumeSpecName: "kube-api-access-whtvj") pod "c4572a53-c77c-4232-9d45-f74d534c98aa" (UID: "c4572a53-c77c-4232-9d45-f74d534c98aa"). InnerVolumeSpecName "kube-api-access-whtvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:56.290804 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:56.290757    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4572a53-c77c-4232-9d45-f74d534c98aa-util" (OuterVolumeSpecName: "util") pod "c4572a53-c77c-4232-9d45-f74d534c98aa" (UID: "c4572a53-c77c-4232-9d45-f74d534c98aa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:56.387334 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:56.387302    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4572a53-c77c-4232-9d45-f74d534c98aa-bundle\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:45:56.387334 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:56.387330    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whtvj\" (UniqueName: \"kubernetes.io/projected/c4572a53-c77c-4232-9d45-f74d534c98aa-kube-api-access-whtvj\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:45:56.387334 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:56.387351    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4572a53-c77c-4232-9d45-f74d534c98aa-util\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:45:56.967015 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:56.966972    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj" event={"ID":"c4572a53-c77c-4232-9d45-f74d534c98aa","Type":"ContainerDied","Data":"82234da06f9d26fb1f59e1a0728828ce1220d2bf02d64127289c93437ca76e24"}
Sep 16 18:45:56.967224 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:56.967037    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82234da06f9d26fb1f59e1a0728828ce1220d2bf02d64127289c93437ca76e24"
Sep 16 18:45:56.967224 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:45:56.966990    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbdgj"
Sep 16 18:46:17.384198 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.384160    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"]
Sep 16 18:46:17.384688 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.384530    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4572a53-c77c-4232-9d45-f74d534c98aa" containerName="util"
Sep 16 18:46:17.384688 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.384544    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4572a53-c77c-4232-9d45-f74d534c98aa" containerName="util"
Sep 16 18:46:17.384688 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.384559    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4572a53-c77c-4232-9d45-f74d534c98aa" containerName="pull"
Sep 16 18:46:17.384688 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.384565    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4572a53-c77c-4232-9d45-f74d534c98aa" containerName="pull"
Sep 16 18:46:17.384688 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.384576    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4572a53-c77c-4232-9d45-f74d534c98aa" containerName="extract"
Sep 16 18:46:17.384688 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.384581    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4572a53-c77c-4232-9d45-f74d534c98aa" containerName="extract"
Sep 16 18:46:17.384688 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.384634    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4572a53-c77c-4232-9d45-f74d534c98aa" containerName="extract"
Sep 16 18:46:17.387653 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.387636    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.390019 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.389991    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:46:17.390191 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.390027    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:46:17.390191 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.390119    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 18:46:17.390191 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.390180    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 18:46:17.390373 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.390310    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 18:46:17.390704 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.390686    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:46:17.390853 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.390837    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ghwh6\""
Sep 16 18:46:17.406531 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.406502    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"]
Sep 16 18:46:17.459002 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.458962    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.459002 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.459002    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.459216 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.459082    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.459216 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.459153    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a740f62-b581-4c0a-8cda-58cb1431e843-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.459216 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.459201    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtfjr\" (UniqueName: \"kubernetes.io/projected/8a740f62-b581-4c0a-8cda-58cb1431e843-kube-api-access-wtfjr\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.459310 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.459219    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8a740f62-b581-4c0a-8cda-58cb1431e843-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.459310 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.459252    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.560651 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.560613    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtfjr\" (UniqueName: \"kubernetes.io/projected/8a740f62-b581-4c0a-8cda-58cb1431e843-kube-api-access-wtfjr\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.560651 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.560658    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8a740f62-b581-4c0a-8cda-58cb1431e843-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.560907 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.560691    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.560907 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.560712    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.560907 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.560731    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.560907 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.560763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.560907 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.560791    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a740f62-b581-4c0a-8cda-58cb1431e843-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.561450 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.561427    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.563317 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.563290    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.563443 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.563388    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.563600 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.563577    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a740f62-b581-4c0a-8cda-58cb1431e843-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.563680 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.563658    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8a740f62-b581-4c0a-8cda-58cb1431e843-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.576913 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.576885    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtfjr\" (UniqueName: \"kubernetes.io/projected/8a740f62-b581-4c0a-8cda-58cb1431e843-kube-api-access-wtfjr\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.588303 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.588274    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9bnkb\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.697510 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.697422    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:17.980097 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:17.980061    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"]
Sep 16 18:46:17.986000 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:46:17.985931    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a740f62_b581_4c0a_8cda_58cb1431e843.slice/crio-69c08e2718e986086eeaab6471f2efd64495ab95898017512552adccef0c5f61 WatchSource:0}: Error finding container 69c08e2718e986086eeaab6471f2efd64495ab95898017512552adccef0c5f61: Status 404 returned error can't find the container with id 69c08e2718e986086eeaab6471f2efd64495ab95898017512552adccef0c5f61
Sep 16 18:46:18.060809 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:18.060777    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb" event={"ID":"8a740f62-b581-4c0a-8cda-58cb1431e843","Type":"ContainerStarted","Data":"69c08e2718e986086eeaab6471f2efd64495ab95898017512552adccef0c5f61"}
Sep 16 18:46:20.398711 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:20.398669    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:46:20.398989 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:20.398740    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:46:21.086772 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:21.086721    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb" event={"ID":"8a740f62-b581-4c0a-8cda-58cb1431e843","Type":"ContainerStarted","Data":"ba258de62594b371c9ee4cb872531e8ac54b1f786f18fb2e6cc2131854790197"}
Sep 16 18:46:21.087288 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:21.087257    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:21.089447 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:21.089412    2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-9bnkb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 18:46:21.089595 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:21.089488    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb" podUID="8a740f62-b581-4c0a-8cda-58cb1431e843" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:46:21.116522 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:21.116460    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb" podStartSLOduration=1.705586203 podStartE2EDuration="4.1164424s" totalImagesPullingTime="2.410856197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:21.1157721 +0000 UTC m=+419.475290095" watchObservedRunningTime="2026-09-16 18:46:21.1164424 +0000 UTC m=+419.475960395"
Sep 16 18:46:22.092596 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:22.092566    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:46:34.143181 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.143135    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j"]
Sep 16 18:46:34.151261 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.151232    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j"
Sep 16 18:46:34.153978 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.153953    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-55b4r\""
Sep 16 18:46:34.154126 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.153996    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:46:34.154858 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.154837    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:46:34.159777 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.159751    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j"]
Sep 16 18:46:34.216025 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.215979    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d480a1cc-dc07-4552-8e17-54df7e02edf3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j\" (UID: \"d480a1cc-dc07-4552-8e17-54df7e02edf3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j"
Sep 16 18:46:34.216182 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.216039    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj2v7\" (UniqueName: \"kubernetes.io/projected/d480a1cc-dc07-4552-8e17-54df7e02edf3-kube-api-access-mj2v7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j\" (UID: \"d480a1cc-dc07-4552-8e17-54df7e02edf3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j"
Sep 16 18:46:34.216182 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.216085    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d480a1cc-dc07-4552-8e17-54df7e02edf3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j\" (UID: \"d480a1cc-dc07-4552-8e17-54df7e02edf3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j"
Sep 16 18:46:34.255159 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.255122    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt"]
Sep 16 18:46:34.258973 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.258950    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt"
Sep 16 18:46:34.284477 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.284446    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt"]
Sep 16 18:46:34.316947 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.316896    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj2v7\" (UniqueName: \"kubernetes.io/projected/d480a1cc-dc07-4552-8e17-54df7e02edf3-kube-api-access-mj2v7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j\" (UID: \"d480a1cc-dc07-4552-8e17-54df7e02edf3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j"
Sep 16 18:46:34.316947 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.316949    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d480a1cc-dc07-4552-8e17-54df7e02edf3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j\" (UID: \"d480a1cc-dc07-4552-8e17-54df7e02edf3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j"
Sep 16 18:46:34.317431 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.317012    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d480a1cc-dc07-4552-8e17-54df7e02edf3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j\" (UID: \"d480a1cc-dc07-4552-8e17-54df7e02edf3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j"
Sep 16 18:46:34.317431 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.317379    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d480a1cc-dc07-4552-8e17-54df7e02edf3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j\" (UID: \"d480a1cc-dc07-4552-8e17-54df7e02edf3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j"
Sep 16 18:46:34.317431 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.317409    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d480a1cc-dc07-4552-8e17-54df7e02edf3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j\" (UID: \"d480a1cc-dc07-4552-8e17-54df7e02edf3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j"
Sep 16 18:46:34.327225 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.327192    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj2v7\" (UniqueName: \"kubernetes.io/projected/d480a1cc-dc07-4552-8e17-54df7e02edf3-kube-api-access-mj2v7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j\" (UID: \"d480a1cc-dc07-4552-8e17-54df7e02edf3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j"
Sep 16 18:46:34.417640 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.417538    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c08ddbf6-c81f-490f-a247-582474a5dc9f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt\" (UID: \"c08ddbf6-c81f-490f-a247-582474a5dc9f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt"
Sep 16 18:46:34.417640 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.417616    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hlls\" (UniqueName: \"kubernetes.io/projected/c08ddbf6-c81f-490f-a247-582474a5dc9f-kube-api-access-5hlls\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt\" (UID: \"c08ddbf6-c81f-490f-a247-582474a5dc9f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt"
Sep 16 18:46:34.417848 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.417655    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c08ddbf6-c81f-490f-a247-582474a5dc9f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt\" (UID: \"c08ddbf6-c81f-490f-a247-582474a5dc9f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt"
Sep 16 18:46:34.418463 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.418438    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf"]
Sep 16 18:46:34.422354 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.422329    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf"
Sep 16 18:46:34.446176 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.446137    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf"]
Sep 16 18:46:34.462071 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.462040    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j"
Sep 16 18:46:34.519072 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.519036    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlls\" (UniqueName: \"kubernetes.io/projected/c08ddbf6-c81f-490f-a247-582474a5dc9f-kube-api-access-5hlls\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt\" (UID: \"c08ddbf6-c81f-490f-a247-582474a5dc9f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt"
Sep 16 18:46:34.519236 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.519115    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c08ddbf6-c81f-490f-a247-582474a5dc9f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt\" (UID: \"c08ddbf6-c81f-490f-a247-582474a5dc9f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt"
Sep 16 18:46:34.519236 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.519190    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf\" (UID: \"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf"
Sep 16 18:46:34.519371 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.519238    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf\" (UID: \"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf"
Sep 16 18:46:34.519371 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.519260    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c08ddbf6-c81f-490f-a247-582474a5dc9f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt\" (UID: \"c08ddbf6-c81f-490f-a247-582474a5dc9f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt"
Sep 16 18:46:34.519371 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.519289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjrwq\" (UniqueName: \"kubernetes.io/projected/ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2-kube-api-access-xjrwq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf\" (UID: \"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf"
Sep 16 18:46:34.519643 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.519616    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c08ddbf6-c81f-490f-a247-582474a5dc9f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt\" (UID: \"c08ddbf6-c81f-490f-a247-582474a5dc9f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt"
Sep 16 18:46:34.519735 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.519628    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c08ddbf6-c81f-490f-a247-582474a5dc9f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt\" (UID: \"c08ddbf6-c81f-490f-a247-582474a5dc9f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt"
Sep 16 18:46:34.531261 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.531226    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlls\" (UniqueName: \"kubernetes.io/projected/c08ddbf6-c81f-490f-a247-582474a5dc9f-kube-api-access-5hlls\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt\" (UID: \"c08ddbf6-c81f-490f-a247-582474a5dc9f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt"
Sep 16 18:46:34.554639 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.554600    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh"]
Sep 16 18:46:34.559678 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.559652    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh"
Sep 16 18:46:34.568000 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.567973    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt"
Sep 16 18:46:34.577208 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.577178    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh"]
Sep 16 18:46:34.620565 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.620530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf\" (UID: \"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf"
Sep 16 18:46:34.620721 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.620624    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf\" (UID: \"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf"
Sep 16 18:46:34.620721 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.620677    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjrwq\" (UniqueName: \"kubernetes.io/projected/ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2-kube-api-access-xjrwq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf\" (UID: \"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf"
Sep 16 18:46:34.621530 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.621265    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf\" (UID: \"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf"
Sep 16 18:46:34.622669 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.622646    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf\" (UID: \"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf"
Sep 16 18:46:34.630981 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.630946    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjrwq\" (UniqueName: \"kubernetes.io/projected/ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2-kube-api-access-xjrwq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf\" (UID: \"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf"
Sep 16 18:46:34.721699 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.721608    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51690747-308f-4683-b4d7-0f66132784c2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh\" (UID: \"51690747-308f-4683-b4d7-0f66132784c2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh"
Sep 16 18:46:34.721699 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.721663    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51690747-308f-4683-b4d7-0f66132784c2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh\" (UID: \"51690747-308f-4683-b4d7-0f66132784c2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh"
Sep 16 18:46:34.721869 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.721699    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwkfl\" (UniqueName: \"kubernetes.io/projected/51690747-308f-4683-b4d7-0f66132784c2-kube-api-access-nwkfl\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh\" (UID: \"51690747-308f-4683-b4d7-0f66132784c2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh"
Sep 16 18:46:34.732512 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.732472    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf"
Sep 16 18:46:34.740253 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.740225    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j"]
Sep 16 18:46:34.746156 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:46:34.746123    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd480a1cc_dc07_4552_8e17_54df7e02edf3.slice/crio-2a81e54868dfdc823bbb52e89d7bd92eea42cf26b872a8a71ce2e0e460f95394 WatchSource:0}: Error finding container 2a81e54868dfdc823bbb52e89d7bd92eea42cf26b872a8a71ce2e0e460f95394: Status 404 returned error can't find the container with id 2a81e54868dfdc823bbb52e89d7bd92eea42cf26b872a8a71ce2e0e460f95394
Sep 16 18:46:34.823110 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.823055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51690747-308f-4683-b4d7-0f66132784c2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh\" (UID: \"51690747-308f-4683-b4d7-0f66132784c2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh"
Sep 16 18:46:34.823297 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.823168    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51690747-308f-4683-b4d7-0f66132784c2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh\" (UID: \"51690747-308f-4683-b4d7-0f66132784c2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh"
Sep 16 18:46:34.823297 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.823232    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwkfl\" (UniqueName: \"kubernetes.io/projected/51690747-308f-4683-b4d7-0f66132784c2-kube-api-access-nwkfl\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh\" (UID: \"51690747-308f-4683-b4d7-0f66132784c2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh"
Sep 16 18:46:34.823573 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.823550    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51690747-308f-4683-b4d7-0f66132784c2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh\" (UID: \"51690747-308f-4683-b4d7-0f66132784c2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh"
Sep 16 18:46:34.823683 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.823665    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51690747-308f-4683-b4d7-0f66132784c2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh\" (UID: \"51690747-308f-4683-b4d7-0f66132784c2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh"
Sep 16 18:46:34.833097 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.833066    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwkfl\" (UniqueName: \"kubernetes.io/projected/51690747-308f-4683-b4d7-0f66132784c2-kube-api-access-nwkfl\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh\" (UID: \"51690747-308f-4683-b4d7-0f66132784c2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh"
Sep 16 18:46:34.851029 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.851001    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt"]
Sep 16 18:46:34.868801 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:34.868758    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh"
Sep 16 18:46:34.905530 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:46:34.905491    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08ddbf6_c81f_490f_a247_582474a5dc9f.slice/crio-95b683ac20a670e1146fb6691e98f6d402293b693c96b2b90e551d10bdeaa007 WatchSource:0}: Error finding container 95b683ac20a670e1146fb6691e98f6d402293b693c96b2b90e551d10bdeaa007: Status 404 returned error can't find the container with id 95b683ac20a670e1146fb6691e98f6d402293b693c96b2b90e551d10bdeaa007
Sep 16 18:46:35.019158 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:35.019120    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf"]
Sep 16 18:46:35.020072 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:46:35.020038    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0a4be2_4b56_4ef2_90d7_4ac3b1e5f3a2.slice/crio-edd8815948ee991cab3f61e9cb1bcdad7aadb458477d937a9ec682017d1a91f7 WatchSource:0}: Error finding container edd8815948ee991cab3f61e9cb1bcdad7aadb458477d937a9ec682017d1a91f7: Status 404 returned error can't find the container with id edd8815948ee991cab3f61e9cb1bcdad7aadb458477d937a9ec682017d1a91f7
Sep 16 18:46:35.153813 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:35.153772    2565 generic.go:356] "Generic (PLEG): container finished" podID="d480a1cc-dc07-4552-8e17-54df7e02edf3" containerID="0c29e1877b71a4d395d81bc8909a575981384ed30430d76f4019b0177fb2946a" exitCode=0
Sep 16 18:46:35.154311 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:35.153854    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j" event={"ID":"d480a1cc-dc07-4552-8e17-54df7e02edf3","Type":"ContainerDied","Data":"0c29e1877b71a4d395d81bc8909a575981384ed30430d76f4019b0177fb2946a"}
Sep 16 18:46:35.154311 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:35.153921    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j" event={"ID":"d480a1cc-dc07-4552-8e17-54df7e02edf3","Type":"ContainerStarted","Data":"2a81e54868dfdc823bbb52e89d7bd92eea42cf26b872a8a71ce2e0e460f95394"}
Sep 16 18:46:35.156683 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:35.156649    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf" event={"ID":"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2","Type":"ContainerStarted","Data":"cd0c132af183f6165679cb451b61ee31eb8441ee66f7ce9171d0c010415b4b56"}
Sep 16 18:46:35.156801 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:35.156707    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf" event={"ID":"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2","Type":"ContainerStarted","Data":"edd8815948ee991cab3f61e9cb1bcdad7aadb458477d937a9ec682017d1a91f7"}
Sep 16 18:46:35.158177 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:35.158149    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh"]
Sep 16 18:46:35.158648 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:35.158623    2565 generic.go:356] "Generic (PLEG): container finished" podID="c08ddbf6-c81f-490f-a247-582474a5dc9f" containerID="5e15d887691c3173cf220b33fd7968a08e0bc04562e9aa3ab80d063b7ba4e7ad" exitCode=0
Sep 16 18:46:35.158697 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:35.158677    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt" event={"ID":"c08ddbf6-c81f-490f-a247-582474a5dc9f","Type":"ContainerDied","Data":"5e15d887691c3173cf220b33fd7968a08e0bc04562e9aa3ab80d063b7ba4e7ad"}
Sep 16 18:46:35.158738 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:35.158711    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt" event={"ID":"c08ddbf6-c81f-490f-a247-582474a5dc9f","Type":"ContainerStarted","Data":"95b683ac20a670e1146fb6691e98f6d402293b693c96b2b90e551d10bdeaa007"}
Sep 16 18:46:35.190240 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:46:35.190204    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51690747_308f_4683_b4d7_0f66132784c2.slice/crio-372b63f3f8a0ea1d2a22f5cbf83c0817684c4e328b382cca692d27cbda921029 WatchSource:0}: Error finding container 372b63f3f8a0ea1d2a22f5cbf83c0817684c4e328b382cca692d27cbda921029: Status 404 returned error can't find the container with id 372b63f3f8a0ea1d2a22f5cbf83c0817684c4e328b382cca692d27cbda921029
Sep 16 18:46:36.167389 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:36.167320    2565 generic.go:356] "Generic (PLEG): container finished" podID="ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2" containerID="cd0c132af183f6165679cb451b61ee31eb8441ee66f7ce9171d0c010415b4b56" exitCode=0
Sep 16 18:46:36.167845 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:36.167406    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf" event={"ID":"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2","Type":"ContainerDied","Data":"cd0c132af183f6165679cb451b61ee31eb8441ee66f7ce9171d0c010415b4b56"}
Sep 16 18:46:36.169161 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:36.169088    2565 generic.go:356] "Generic (PLEG): container finished" podID="51690747-308f-4683-b4d7-0f66132784c2" containerID="8888cb8ca103bebdbec313c1178c57be262bdfe882d97445f016a27e18eef907" exitCode=0
Sep 16 18:46:36.169230 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:36.169162    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh" event={"ID":"51690747-308f-4683-b4d7-0f66132784c2","Type":"ContainerDied","Data":"8888cb8ca103bebdbec313c1178c57be262bdfe882d97445f016a27e18eef907"}
Sep 16 18:46:36.169230 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:36.169202    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh" event={"ID":"51690747-308f-4683-b4d7-0f66132784c2","Type":"ContainerStarted","Data":"372b63f3f8a0ea1d2a22f5cbf83c0817684c4e328b382cca692d27cbda921029"}
Sep 16 18:46:37.176975 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:37.176937    2565 generic.go:356] "Generic (PLEG): container finished" podID="d480a1cc-dc07-4552-8e17-54df7e02edf3" containerID="efe7e19806a777e150332ceca82239ad1534865b13187d2dad282465b150f19e" exitCode=0
Sep 16 18:46:37.177502 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:37.176996    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j" event={"ID":"d480a1cc-dc07-4552-8e17-54df7e02edf3","Type":"ContainerDied","Data":"efe7e19806a777e150332ceca82239ad1534865b13187d2dad282465b150f19e"}
Sep 16 18:46:37.179158 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:37.179126    2565 generic.go:356] "Generic (PLEG): container finished" podID="ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2" containerID="af5ff3c81f8a1da75928838aadf9e26b02a0e21fce690a4173321abfe22c5caa" exitCode=0
Sep 16 18:46:37.179250 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:37.179212    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf" event={"ID":"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2","Type":"ContainerDied","Data":"af5ff3c81f8a1da75928838aadf9e26b02a0e21fce690a4173321abfe22c5caa"}
Sep 16 18:46:37.181378 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:37.181334    2565 generic.go:356] "Generic (PLEG): container finished" podID="c08ddbf6-c81f-490f-a247-582474a5dc9f" containerID="6634c2116fd1772accb8ca35a0bb1d296c774f033fad611fdbbccaf03f41922d" exitCode=0
Sep 16 18:46:37.181454 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:37.181377    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt" event={"ID":"c08ddbf6-c81f-490f-a247-582474a5dc9f","Type":"ContainerDied","Data":"6634c2116fd1772accb8ca35a0bb1d296c774f033fad611fdbbccaf03f41922d"}
Sep 16 18:46:38.189711 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:38.189669    2565 generic.go:356] "Generic (PLEG): container finished" podID="ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2" containerID="8d2566dfaf10984540172ae3bd280585aa1a5528ca8f2ca8a9592372f80ffffb" exitCode=0
Sep 16 18:46:38.190115 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:38.189764    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf" event={"ID":"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2","Type":"ContainerDied","Data":"8d2566dfaf10984540172ae3bd280585aa1a5528ca8f2ca8a9592372f80ffffb"}
Sep 16 18:46:38.191974 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:38.191946    2565 generic.go:356] "Generic (PLEG): container finished" podID="c08ddbf6-c81f-490f-a247-582474a5dc9f" containerID="b506918d0d623c7fcbbbfbbdb6c4bf9dbd7d85545886293f0fcea39feeeee682" exitCode=0
Sep 16 18:46:38.192097 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:38.192032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt" event={"ID":"c08ddbf6-c81f-490f-a247-582474a5dc9f","Type":"ContainerDied","Data":"b506918d0d623c7fcbbbfbbdb6c4bf9dbd7d85545886293f0fcea39feeeee682"}
Sep 16 18:46:38.194058 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:38.194021    2565 generic.go:356] "Generic (PLEG): container finished" podID="51690747-308f-4683-b4d7-0f66132784c2" containerID="1a3c21b1497b9ea8dca0bf884669a2f0675a5f3dbb70f2969dd9f83cca1e387f" exitCode=0
Sep 16 18:46:38.194190 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:38.194111    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh" event={"ID":"51690747-308f-4683-b4d7-0f66132784c2","Type":"ContainerDied","Data":"1a3c21b1497b9ea8dca0bf884669a2f0675a5f3dbb70f2969dd9f83cca1e387f"}
Sep 16 18:46:38.196461 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:38.196442    2565 generic.go:356] "Generic (PLEG): container finished" podID="d480a1cc-dc07-4552-8e17-54df7e02edf3" containerID="9a10d66c20f6bb61375b7a5c0d90888682caabd26913d460f0cbfe4a8063fe3d" exitCode=0
Sep 16 18:46:38.196557 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:38.196493    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j" event={"ID":"d480a1cc-dc07-4552-8e17-54df7e02edf3","Type":"ContainerDied","Data":"9a10d66c20f6bb61375b7a5c0d90888682caabd26913d460f0cbfe4a8063fe3d"}
Sep 16 18:46:39.204430 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.204395    2565 generic.go:356] "Generic (PLEG): container finished" podID="51690747-308f-4683-b4d7-0f66132784c2" containerID="bffe4b195ee72a28c3f13fb38c2deb0cc998a365a205c07c872005d72730e0ab" exitCode=0
Sep 16 18:46:39.204943 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.204472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh" event={"ID":"51690747-308f-4683-b4d7-0f66132784c2","Type":"ContainerDied","Data":"bffe4b195ee72a28c3f13fb38c2deb0cc998a365a205c07c872005d72730e0ab"}
Sep 16 18:46:39.392116 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.392083    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf"
Sep 16 18:46:39.396103 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.396066    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt"
Sep 16 18:46:39.414193 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.414164    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j"
Sep 16 18:46:39.575217 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.575154    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d480a1cc-dc07-4552-8e17-54df7e02edf3-util\") pod \"d480a1cc-dc07-4552-8e17-54df7e02edf3\" (UID: \"d480a1cc-dc07-4552-8e17-54df7e02edf3\") "
Sep 16 18:46:39.575449 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.575236    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hlls\" (UniqueName: \"kubernetes.io/projected/c08ddbf6-c81f-490f-a247-582474a5dc9f-kube-api-access-5hlls\") pod \"c08ddbf6-c81f-490f-a247-582474a5dc9f\" (UID: \"c08ddbf6-c81f-490f-a247-582474a5dc9f\") "
Sep 16 18:46:39.575449 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.575276    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjrwq\" (UniqueName: \"kubernetes.io/projected/ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2-kube-api-access-xjrwq\") pod \"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2\" (UID: \"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2\") "
Sep 16 18:46:39.575449 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.575304    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2-util\") pod \"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2\" (UID: \"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2\") "
Sep 16 18:46:39.575449 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.575396    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj2v7\" (UniqueName: \"kubernetes.io/projected/d480a1cc-dc07-4552-8e17-54df7e02edf3-kube-api-access-mj2v7\") pod \"d480a1cc-dc07-4552-8e17-54df7e02edf3\" (UID: \"d480a1cc-dc07-4552-8e17-54df7e02edf3\") "
Sep 16 18:46:39.575449 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.575422    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c08ddbf6-c81f-490f-a247-582474a5dc9f-bundle\") pod \"c08ddbf6-c81f-490f-a247-582474a5dc9f\" (UID: \"c08ddbf6-c81f-490f-a247-582474a5dc9f\") "
Sep 16 18:46:39.575682 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.575472    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c08ddbf6-c81f-490f-a247-582474a5dc9f-util\") pod \"c08ddbf6-c81f-490f-a247-582474a5dc9f\" (UID: \"c08ddbf6-c81f-490f-a247-582474a5dc9f\") "
Sep 16 18:46:39.575682 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.575506    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2-bundle\") pod \"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2\" (UID: \"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2\") "
Sep 16 18:46:39.575682 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.575539    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d480a1cc-dc07-4552-8e17-54df7e02edf3-bundle\") pod \"d480a1cc-dc07-4552-8e17-54df7e02edf3\" (UID: \"d480a1cc-dc07-4552-8e17-54df7e02edf3\") "
Sep 16 18:46:39.576069 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.576009    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08ddbf6-c81f-490f-a247-582474a5dc9f-bundle" (OuterVolumeSpecName: "bundle") pod "c08ddbf6-c81f-490f-a247-582474a5dc9f" (UID: "c08ddbf6-c81f-490f-a247-582474a5dc9f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:39.576217 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.576188    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2-bundle" (OuterVolumeSpecName: "bundle") pod "ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2" (UID: "ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:39.576291 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.576241    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d480a1cc-dc07-4552-8e17-54df7e02edf3-bundle" (OuterVolumeSpecName: "bundle") pod "d480a1cc-dc07-4552-8e17-54df7e02edf3" (UID: "d480a1cc-dc07-4552-8e17-54df7e02edf3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:39.577986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.577947    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d480a1cc-dc07-4552-8e17-54df7e02edf3-kube-api-access-mj2v7" (OuterVolumeSpecName: "kube-api-access-mj2v7") pod "d480a1cc-dc07-4552-8e17-54df7e02edf3" (UID: "d480a1cc-dc07-4552-8e17-54df7e02edf3"). InnerVolumeSpecName "kube-api-access-mj2v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:39.578108 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.578086    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08ddbf6-c81f-490f-a247-582474a5dc9f-kube-api-access-5hlls" (OuterVolumeSpecName: "kube-api-access-5hlls") pod "c08ddbf6-c81f-490f-a247-582474a5dc9f" (UID: "c08ddbf6-c81f-490f-a247-582474a5dc9f"). InnerVolumeSpecName "kube-api-access-5hlls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:39.578631 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.578614    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2-kube-api-access-xjrwq" (OuterVolumeSpecName: "kube-api-access-xjrwq") pod "ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2" (UID: "ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2"). InnerVolumeSpecName "kube-api-access-xjrwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:39.581937 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.581893    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d480a1cc-dc07-4552-8e17-54df7e02edf3-util" (OuterVolumeSpecName: "util") pod "d480a1cc-dc07-4552-8e17-54df7e02edf3" (UID: "d480a1cc-dc07-4552-8e17-54df7e02edf3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:39.582026 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.581970    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08ddbf6-c81f-490f-a247-582474a5dc9f-util" (OuterVolumeSpecName: "util") pod "c08ddbf6-c81f-490f-a247-582474a5dc9f" (UID: "c08ddbf6-c81f-490f-a247-582474a5dc9f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:39.582103 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.582084    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2-util" (OuterVolumeSpecName: "util") pod "ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2" (UID: "ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:39.676589 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.676535    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj2v7\" (UniqueName: \"kubernetes.io/projected/d480a1cc-dc07-4552-8e17-54df7e02edf3-kube-api-access-mj2v7\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:46:39.676589 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.676582    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c08ddbf6-c81f-490f-a247-582474a5dc9f-bundle\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:46:39.676589 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.676594    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c08ddbf6-c81f-490f-a247-582474a5dc9f-util\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:46:39.676589 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.676602    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2-bundle\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:46:39.676589 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.676609    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d480a1cc-dc07-4552-8e17-54df7e02edf3-bundle\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:46:39.676900 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.676616    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d480a1cc-dc07-4552-8e17-54df7e02edf3-util\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:46:39.676900 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.676626    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hlls\" (UniqueName: \"kubernetes.io/projected/c08ddbf6-c81f-490f-a247-582474a5dc9f-kube-api-access-5hlls\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:46:39.676900 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.676635    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjrwq\" (UniqueName: \"kubernetes.io/projected/ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2-kube-api-access-xjrwq\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:46:39.676900 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:39.676645    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2-util\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:46:40.058011 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.057974    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-689959bb58-c2k5g"]
Sep 16 18:46:40.058336 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058324    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d480a1cc-dc07-4552-8e17-54df7e02edf3" containerName="extract"
Sep 16 18:46:40.058405 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058338    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d480a1cc-dc07-4552-8e17-54df7e02edf3" containerName="extract"
Sep 16 18:46:40.058405 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058363    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c08ddbf6-c81f-490f-a247-582474a5dc9f" containerName="extract"
Sep 16 18:46:40.058405 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058368    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c08ddbf6-c81f-490f-a247-582474a5dc9f" containerName="extract"
Sep 16 18:46:40.058405 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058381    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2" containerName="pull"
Sep 16 18:46:40.058405 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058387    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2" containerName="pull"
Sep 16 18:46:40.058405 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058394    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2" containerName="extract"
Sep 16 18:46:40.058405 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058399    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2" containerName="extract"
Sep 16 18:46:40.058405 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058408    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d480a1cc-dc07-4552-8e17-54df7e02edf3" containerName="util"
Sep 16 18:46:40.058642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058413    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d480a1cc-dc07-4552-8e17-54df7e02edf3" containerName="util"
Sep 16 18:46:40.058642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058425    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c08ddbf6-c81f-490f-a247-582474a5dc9f" containerName="util"
Sep 16 18:46:40.058642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058431    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c08ddbf6-c81f-490f-a247-582474a5dc9f" containerName="util"
Sep 16 18:46:40.058642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058437    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2" containerName="util"
Sep 16 18:46:40.058642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058442    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2" containerName="util"
Sep 16 18:46:40.058642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058451    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c08ddbf6-c81f-490f-a247-582474a5dc9f" containerName="pull"
Sep 16 18:46:40.058642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058456    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c08ddbf6-c81f-490f-a247-582474a5dc9f" containerName="pull"
Sep 16 18:46:40.058642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058462    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d480a1cc-dc07-4552-8e17-54df7e02edf3" containerName="pull"
Sep 16 18:46:40.058642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058466    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d480a1cc-dc07-4552-8e17-54df7e02edf3" containerName="pull"
Sep 16 18:46:40.058642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058514    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d480a1cc-dc07-4552-8e17-54df7e02edf3" containerName="extract"
Sep 16 18:46:40.058642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058524    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2" containerName="extract"
Sep 16 18:46:40.058642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.058532    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c08ddbf6-c81f-490f-a247-582474a5dc9f" containerName="extract"
Sep 16 18:46:40.062881 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.062854    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.075483 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.075451    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-689959bb58-c2k5g"]
Sep 16 18:46:40.181595 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.181536    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b609927-192c-465e-b5a2-07b92efee24c-service-ca\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.181595 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.181594    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t895t\" (UniqueName: \"kubernetes.io/projected/1b609927-192c-465e-b5a2-07b92efee24c-kube-api-access-t895t\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.181834 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.181683    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b609927-192c-465e-b5a2-07b92efee24c-console-oauth-config\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.181834 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.181717    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b609927-192c-465e-b5a2-07b92efee24c-console-serving-cert\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.181834 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.181752    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b609927-192c-465e-b5a2-07b92efee24c-console-config\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.181834 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.181775    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b609927-192c-465e-b5a2-07b92efee24c-oauth-serving-cert\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.181834 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.181821    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b609927-192c-465e-b5a2-07b92efee24c-trusted-ca-bundle\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.213237 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.213200    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j"
Sep 16 18:46:40.213738 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.213200    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ht9j" event={"ID":"d480a1cc-dc07-4552-8e17-54df7e02edf3","Type":"ContainerDied","Data":"2a81e54868dfdc823bbb52e89d7bd92eea42cf26b872a8a71ce2e0e460f95394"}
Sep 16 18:46:40.213738 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.213335    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a81e54868dfdc823bbb52e89d7bd92eea42cf26b872a8a71ce2e0e460f95394"
Sep 16 18:46:40.215492 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.215467    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf"
Sep 16 18:46:40.215666 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.215486    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arvcdf" event={"ID":"ce0a4be2-4b56-4ef2-90d7-4ac3b1e5f3a2","Type":"ContainerDied","Data":"edd8815948ee991cab3f61e9cb1bcdad7aadb458477d937a9ec682017d1a91f7"}
Sep 16 18:46:40.215666 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.215537    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edd8815948ee991cab3f61e9cb1bcdad7aadb458477d937a9ec682017d1a91f7"
Sep 16 18:46:40.217994 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.217969    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt"
Sep 16 18:46:40.217994 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.217987    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8rt" event={"ID":"c08ddbf6-c81f-490f-a247-582474a5dc9f","Type":"ContainerDied","Data":"95b683ac20a670e1146fb6691e98f6d402293b693c96b2b90e551d10bdeaa007"}
Sep 16 18:46:40.218168 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.218026    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95b683ac20a670e1146fb6691e98f6d402293b693c96b2b90e551d10bdeaa007"
Sep 16 18:46:40.283047 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.283005    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b609927-192c-465e-b5a2-07b92efee24c-service-ca\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.283047 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.283049    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t895t\" (UniqueName: \"kubernetes.io/projected/1b609927-192c-465e-b5a2-07b92efee24c-kube-api-access-t895t\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.283452 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.283088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b609927-192c-465e-b5a2-07b92efee24c-console-oauth-config\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.283452 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.283109    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b609927-192c-465e-b5a2-07b92efee24c-console-serving-cert\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.283452 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.283144    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b609927-192c-465e-b5a2-07b92efee24c-console-config\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.283452 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.283321    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b609927-192c-465e-b5a2-07b92efee24c-oauth-serving-cert\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.283452 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.283413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b609927-192c-465e-b5a2-07b92efee24c-trusted-ca-bundle\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.283952 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.283919    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b609927-192c-465e-b5a2-07b92efee24c-service-ca\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.284374 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.284306    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b609927-192c-465e-b5a2-07b92efee24c-trusted-ca-bundle\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.284482 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.284445    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b609927-192c-465e-b5a2-07b92efee24c-console-config\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.284482 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.284461    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b609927-192c-465e-b5a2-07b92efee24c-oauth-serving-cert\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.286414 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.286391    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b609927-192c-465e-b5a2-07b92efee24c-console-serving-cert\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.286695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.286677    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b609927-192c-465e-b5a2-07b92efee24c-console-oauth-config\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.292405 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.292380    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t895t\" (UniqueName: \"kubernetes.io/projected/1b609927-192c-465e-b5a2-07b92efee24c-kube-api-access-t895t\") pod \"console-689959bb58-c2k5g\" (UID: \"1b609927-192c-465e-b5a2-07b92efee24c\") " pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.342178 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.342148    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh"
Sep 16 18:46:40.372830 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.372788    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:40.485823 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.485784    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51690747-308f-4683-b4d7-0f66132784c2-bundle\") pod \"51690747-308f-4683-b4d7-0f66132784c2\" (UID: \"51690747-308f-4683-b4d7-0f66132784c2\") "
Sep 16 18:46:40.486002 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.485847    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51690747-308f-4683-b4d7-0f66132784c2-util\") pod \"51690747-308f-4683-b4d7-0f66132784c2\" (UID: \"51690747-308f-4683-b4d7-0f66132784c2\") "
Sep 16 18:46:40.486002 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.485876    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwkfl\" (UniqueName: \"kubernetes.io/projected/51690747-308f-4683-b4d7-0f66132784c2-kube-api-access-nwkfl\") pod \"51690747-308f-4683-b4d7-0f66132784c2\" (UID: \"51690747-308f-4683-b4d7-0f66132784c2\") "
Sep 16 18:46:40.486522 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.486477    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51690747-308f-4683-b4d7-0f66132784c2-bundle" (OuterVolumeSpecName: "bundle") pod "51690747-308f-4683-b4d7-0f66132784c2" (UID: "51690747-308f-4683-b4d7-0f66132784c2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:40.488173 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.488142    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51690747-308f-4683-b4d7-0f66132784c2-kube-api-access-nwkfl" (OuterVolumeSpecName: "kube-api-access-nwkfl") pod "51690747-308f-4683-b4d7-0f66132784c2" (UID: "51690747-308f-4683-b4d7-0f66132784c2"). InnerVolumeSpecName "kube-api-access-nwkfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:40.491282 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.491258    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51690747-308f-4683-b4d7-0f66132784c2-util" (OuterVolumeSpecName: "util") pod "51690747-308f-4683-b4d7-0f66132784c2" (UID: "51690747-308f-4683-b4d7-0f66132784c2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:40.586854 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.586764    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwkfl\" (UniqueName: \"kubernetes.io/projected/51690747-308f-4683-b4d7-0f66132784c2-kube-api-access-nwkfl\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:46:40.586854 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.586797    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51690747-308f-4683-b4d7-0f66132784c2-bundle\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:46:40.586854 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.586806    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51690747-308f-4683-b4d7-0f66132784c2-util\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:46:40.653813 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:40.653783    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-689959bb58-c2k5g"]
Sep 16 18:46:40.655450 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:46:40.655422    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b609927_192c_465e_b5a2_07b92efee24c.slice/crio-3eedc05fc1dd26284bfce19c77de4e2c6a8acb4028341da0db3ff818d9a6368a WatchSource:0}: Error finding container 3eedc05fc1dd26284bfce19c77de4e2c6a8acb4028341da0db3ff818d9a6368a: Status 404 returned error can't find the container with id 3eedc05fc1dd26284bfce19c77de4e2c6a8acb4028341da0db3ff818d9a6368a
Sep 16 18:46:41.225164 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:41.225125    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh"
Sep 16 18:46:41.225164 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:41.225147    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdb9rh" event={"ID":"51690747-308f-4683-b4d7-0f66132784c2","Type":"ContainerDied","Data":"372b63f3f8a0ea1d2a22f5cbf83c0817684c4e328b382cca692d27cbda921029"}
Sep 16 18:46:41.225729 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:41.225200    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="372b63f3f8a0ea1d2a22f5cbf83c0817684c4e328b382cca692d27cbda921029"
Sep 16 18:46:41.226871 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:41.226772    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689959bb58-c2k5g" event={"ID":"1b609927-192c-465e-b5a2-07b92efee24c","Type":"ContainerStarted","Data":"715a28d203d5beb4ff60205fd6d8bf472eb8e2f82727b9e95db490894ba7df48"}
Sep 16 18:46:41.226971 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:41.226894    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689959bb58-c2k5g" event={"ID":"1b609927-192c-465e-b5a2-07b92efee24c","Type":"ContainerStarted","Data":"3eedc05fc1dd26284bfce19c77de4e2c6a8acb4028341da0db3ff818d9a6368a"}
Sep 16 18:46:41.253936 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:41.253879    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-689959bb58-c2k5g" podStartSLOduration=1.2538587589999999 podStartE2EDuration="1.253858759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:41.25209482 +0000 UTC m=+439.611612818" watchObservedRunningTime="2026-09-16 18:46:41.253858759 +0000 UTC m=+439.613376769"
Sep 16 18:46:48.710582 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:48.710498    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-hf6t2"]
Sep 16 18:46:48.711024 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:48.710837    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51690747-308f-4683-b4d7-0f66132784c2" containerName="util"
Sep 16 18:46:48.711024 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:48.710848    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="51690747-308f-4683-b4d7-0f66132784c2" containerName="util"
Sep 16 18:46:48.711024 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:48.710858    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51690747-308f-4683-b4d7-0f66132784c2" containerName="extract"
Sep 16 18:46:48.711024 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:48.710865    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="51690747-308f-4683-b4d7-0f66132784c2" containerName="extract"
Sep 16 18:46:48.711024 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:48.710881    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51690747-308f-4683-b4d7-0f66132784c2" containerName="pull"
Sep 16 18:46:48.711024 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:48.710889    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="51690747-308f-4683-b4d7-0f66132784c2" containerName="pull"
Sep 16 18:46:48.711024 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:48.710940    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="51690747-308f-4683-b4d7-0f66132784c2" containerName="extract"
Sep 16 18:46:48.713839 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:48.713820    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-hf6t2"
Sep 16 18:46:48.716281 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:48.716254    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zd2dv\""
Sep 16 18:46:48.716446 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:48.716262    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:46:48.716446 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:48.716299    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:46:48.729260 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:48.729229    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-hf6t2"]
Sep 16 18:46:48.758433 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:48.758396    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjg27\" (UniqueName: \"kubernetes.io/projected/14eeb42f-1bd4-4b63-a462-487e6d1c7187-kube-api-access-tjg27\") pod \"kuadrant-operator-controller-manager-6d4cb79c-hf6t2\" (UID: \"14eeb42f-1bd4-4b63-a462-487e6d1c7187\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-hf6t2"
Sep 16 18:46:48.758619 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:48.758448    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14eeb42f-1bd4-4b63-a462-487e6d1c7187-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-hf6t2\" (UID: \"14eeb42f-1bd4-4b63-a462-487e6d1c7187\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-hf6t2"
Sep 16 18:46:48.859333 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:48.859286    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14eeb42f-1bd4-4b63-a462-487e6d1c7187-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-hf6t2\" (UID: \"14eeb42f-1bd4-4b63-a462-487e6d1c7187\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-hf6t2"
Sep 16 18:46:48.859548 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:48.859448    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjg27\" (UniqueName: \"kubernetes.io/projected/14eeb42f-1bd4-4b63-a462-487e6d1c7187-kube-api-access-tjg27\") pod \"kuadrant-operator-controller-manager-6d4cb79c-hf6t2\" (UID: \"14eeb42f-1bd4-4b63-a462-487e6d1c7187\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-hf6t2"
Sep 16 18:46:48.859817 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:48.859790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/14eeb42f-1bd4-4b63-a462-487e6d1c7187-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-hf6t2\" (UID: \"14eeb42f-1bd4-4b63-a462-487e6d1c7187\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-hf6t2"
Sep 16 18:46:48.871331 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:48.871292    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjg27\" (UniqueName: \"kubernetes.io/projected/14eeb42f-1bd4-4b63-a462-487e6d1c7187-kube-api-access-tjg27\") pod \"kuadrant-operator-controller-manager-6d4cb79c-hf6t2\" (UID: \"14eeb42f-1bd4-4b63-a462-487e6d1c7187\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-hf6t2"
Sep 16 18:46:49.024834 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:49.024793    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-hf6t2"
Sep 16 18:46:49.309967 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:49.309940    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-hf6t2"]
Sep 16 18:46:49.311766 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:46:49.311737    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14eeb42f_1bd4_4b63_a462_487e6d1c7187.slice/crio-b6c8af7cfae7c2e64394a9a7bccb40527a5af143926b3e82147e4b9482ac1d58 WatchSource:0}: Error finding container b6c8af7cfae7c2e64394a9a7bccb40527a5af143926b3e82147e4b9482ac1d58: Status 404 returned error can't find the container with id b6c8af7cfae7c2e64394a9a7bccb40527a5af143926b3e82147e4b9482ac1d58
Sep 16 18:46:50.275543 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:50.275499    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-hf6t2" event={"ID":"14eeb42f-1bd4-4b63-a462-487e6d1c7187","Type":"ContainerStarted","Data":"b6c8af7cfae7c2e64394a9a7bccb40527a5af143926b3e82147e4b9482ac1d58"}
Sep 16 18:46:50.373408 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:50.373370    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:50.373620 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:50.373441    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:50.380061 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:50.380028    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:50.701208 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:50.701105    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tgvjm"]
Sep 16 18:46:50.704953 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:50.704924    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tgvjm"
Sep 16 18:46:50.707824 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:50.707742    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-zh985\""
Sep 16 18:46:50.720146 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:50.720119    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tgvjm"]
Sep 16 18:46:50.777647 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:50.777609    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29q9j\" (UniqueName: \"kubernetes.io/projected/f356ad9e-45a9-4a92-9352-68ede7f9c77b-kube-api-access-29q9j\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-tgvjm\" (UID: \"f356ad9e-45a9-4a92-9352-68ede7f9c77b\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tgvjm"
Sep 16 18:46:50.879210 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:50.879169    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29q9j\" (UniqueName: \"kubernetes.io/projected/f356ad9e-45a9-4a92-9352-68ede7f9c77b-kube-api-access-29q9j\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-tgvjm\" (UID: \"f356ad9e-45a9-4a92-9352-68ede7f9c77b\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tgvjm"
Sep 16 18:46:50.899410 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:50.899372    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29q9j\" (UniqueName: \"kubernetes.io/projected/f356ad9e-45a9-4a92-9352-68ede7f9c77b-kube-api-access-29q9j\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-tgvjm\" (UID: \"f356ad9e-45a9-4a92-9352-68ede7f9c77b\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tgvjm"
Sep 16 18:46:51.020266 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:51.020217    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tgvjm"
Sep 16 18:46:51.288452 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:51.288290    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-689959bb58-c2k5g"
Sep 16 18:46:51.315671 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:51.315636    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tgvjm"]
Sep 16 18:46:51.318732 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:46:51.318702    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf356ad9e_45a9_4a92_9352_68ede7f9c77b.slice/crio-d9fb70681c24de56ebe274d4bf71e887117fdfce5c1fd3ad6b8be8eb61fcf408 WatchSource:0}: Error finding container d9fb70681c24de56ebe274d4bf71e887117fdfce5c1fd3ad6b8be8eb61fcf408: Status 404 returned error can't find the container with id d9fb70681c24de56ebe274d4bf71e887117fdfce5c1fd3ad6b8be8eb61fcf408
Sep 16 18:46:51.468541 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:51.468503    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fbb95445f-hsgwv"]
Sep 16 18:46:52.290142 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:52.290092    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tgvjm" event={"ID":"f356ad9e-45a9-4a92-9352-68ede7f9c77b","Type":"ContainerStarted","Data":"d9fb70681c24de56ebe274d4bf71e887117fdfce5c1fd3ad6b8be8eb61fcf408"}
Sep 16 18:46:53.298461 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:53.298420    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-hf6t2" event={"ID":"14eeb42f-1bd4-4b63-a462-487e6d1c7187","Type":"ContainerStarted","Data":"282bc95406562494f24f9dc237b745cfa1a0e27b5b5c44ce7fe1c4f783e40f28"}
Sep 16 18:46:53.298934 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:53.298538    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-hf6t2"
Sep 16 18:46:53.331766 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:53.331705    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-hf6t2" podStartSLOduration=1.520502945 podStartE2EDuration="5.331687936s" totalImagesPullingTime="3.811184991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:53.329972351 +0000 UTC m=+451.689490346" watchObservedRunningTime="2026-09-16 18:46:53.331687936 +0000 UTC m=+451.691205931"
Sep 16 18:46:56.409080 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:56.409039    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-rkwxr"]
Sep 16 18:46:56.412735 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:56.412712    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-rkwxr"
Sep 16 18:46:56.422518 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:56.422491    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 18:46:56.422618 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:56.422538    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-p6sv2\""
Sep 16 18:46:56.434959 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:56.434927    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-rkwxr"]
Sep 16 18:46:56.532919 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:56.532876    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr7sr\" (UniqueName: \"kubernetes.io/projected/cd86797c-c8c8-4372-a44e-ca40a3c6a884-kube-api-access-hr7sr\") pod \"dns-operator-controller-manager-65dc8f76f5-rkwxr\" (UID: \"cd86797c-c8c8-4372-a44e-ca40a3c6a884\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-rkwxr"
Sep 16 18:46:56.633635 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:56.633592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr7sr\" (UniqueName: \"kubernetes.io/projected/cd86797c-c8c8-4372-a44e-ca40a3c6a884-kube-api-access-hr7sr\") pod \"dns-operator-controller-manager-65dc8f76f5-rkwxr\" (UID: \"cd86797c-c8c8-4372-a44e-ca40a3c6a884\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-rkwxr"
Sep 16 18:46:56.649560 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:56.649535    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr7sr\" (UniqueName: \"kubernetes.io/projected/cd86797c-c8c8-4372-a44e-ca40a3c6a884-kube-api-access-hr7sr\") pod \"dns-operator-controller-manager-65dc8f76f5-rkwxr\" (UID: \"cd86797c-c8c8-4372-a44e-ca40a3c6a884\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-rkwxr"
Sep 16 18:46:56.723164 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:56.723111    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-rkwxr"
Sep 16 18:46:57.000513 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:57.000433    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-rkwxr"]
Sep 16 18:46:57.003886 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:46:57.003851    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd86797c_c8c8_4372_a44e_ca40a3c6a884.slice/crio-c73f091a94ed27cc510c1cdd5be88e19331ee978a12e92de38c612f3b7491615 WatchSource:0}: Error finding container c73f091a94ed27cc510c1cdd5be88e19331ee978a12e92de38c612f3b7491615: Status 404 returned error can't find the container with id c73f091a94ed27cc510c1cdd5be88e19331ee978a12e92de38c612f3b7491615
Sep 16 18:46:57.322599 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:57.322465    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-rkwxr" event={"ID":"cd86797c-c8c8-4372-a44e-ca40a3c6a884","Type":"ContainerStarted","Data":"c73f091a94ed27cc510c1cdd5be88e19331ee978a12e92de38c612f3b7491615"}
Sep 16 18:46:58.329573 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:58.329538    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tgvjm" event={"ID":"f356ad9e-45a9-4a92-9352-68ede7f9c77b","Type":"ContainerStarted","Data":"f16b466fc6270a34f3b5a044a67b6cb9d5c249d706926307f78109b6abf3ab41"}
Sep 16 18:46:58.329968 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:58.329612    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tgvjm"
Sep 16 18:46:58.354902 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:46:58.354845    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tgvjm" podStartSLOduration=1.5305425449999999 podStartE2EDuration="8.354828325s" totalImagesPullingTime="6.82428578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:58.354716766 +0000 UTC m=+456.714234782" watchObservedRunningTime="2026-09-16 18:46:58.354828325 +0000 UTC m=+456.714346319"
Sep 16 18:47:00.342701 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:00.342659    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-rkwxr" event={"ID":"cd86797c-c8c8-4372-a44e-ca40a3c6a884","Type":"ContainerStarted","Data":"5a04713549d6d10713d4ee2f88bd74b1236a6db472a23de188c29eb0c8b825f7"}
Sep 16 18:47:00.343135 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:00.342803    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-rkwxr"
Sep 16 18:47:00.374496 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:00.374433    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-rkwxr" podStartSLOduration=1.466529268 podStartE2EDuration="4.374410231s" totalImagesPullingTime="2.907880963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:00.373696952 +0000 UTC m=+458.733214946" watchObservedRunningTime="2026-09-16 18:47:00.374410231 +0000 UTC m=+458.733928227"
Sep 16 18:47:04.306491 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:04.306460    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-hf6t2"
Sep 16 18:47:09.339222 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:09.339188    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-tgvjm"
Sep 16 18:47:11.352184 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:11.352154    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-rkwxr"
Sep 16 18:47:16.493665 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.493587    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6fbb95445f-hsgwv" podUID="7ca8f66d-25e2-4a13-9476-f731e526e11b" containerName="console" containerID="cri-o://4032062f14ae2f245fb43cb088f80251defa383bba56ca1caa6966225453a031" gracePeriod=15
Sep 16 18:47:16.748543 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.748467    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:47:16.814211 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.814169    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-service-ca\") pod \"7ca8f66d-25e2-4a13-9476-f731e526e11b\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") "
Sep 16 18:47:16.814211 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.814218    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-trusted-ca-bundle\") pod \"7ca8f66d-25e2-4a13-9476-f731e526e11b\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") "
Sep 16 18:47:16.814481 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.814249    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ca8f66d-25e2-4a13-9476-f731e526e11b-console-serving-cert\") pod \"7ca8f66d-25e2-4a13-9476-f731e526e11b\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") "
Sep 16 18:47:16.814481 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.814272    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ca8f66d-25e2-4a13-9476-f731e526e11b-console-oauth-config\") pod \"7ca8f66d-25e2-4a13-9476-f731e526e11b\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") "
Sep 16 18:47:16.814481 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.814447    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbk4v\" (UniqueName: \"kubernetes.io/projected/7ca8f66d-25e2-4a13-9476-f731e526e11b-kube-api-access-fbk4v\") pod \"7ca8f66d-25e2-4a13-9476-f731e526e11b\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") "
Sep 16 18:47:16.814630 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.814562    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-oauth-serving-cert\") pod \"7ca8f66d-25e2-4a13-9476-f731e526e11b\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") "
Sep 16 18:47:16.814630 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.814601    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-console-config\") pod \"7ca8f66d-25e2-4a13-9476-f731e526e11b\" (UID: \"7ca8f66d-25e2-4a13-9476-f731e526e11b\") "
Sep 16 18:47:16.814730 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.814661    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-service-ca" (OuterVolumeSpecName: "service-ca") pod "7ca8f66d-25e2-4a13-9476-f731e526e11b" (UID: "7ca8f66d-25e2-4a13-9476-f731e526e11b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:16.814730 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.814710    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7ca8f66d-25e2-4a13-9476-f731e526e11b" (UID: "7ca8f66d-25e2-4a13-9476-f731e526e11b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:16.814900 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.814878    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-service-ca\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:47:16.814951 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.814902    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-trusted-ca-bundle\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:47:16.815049 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.815019    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7ca8f66d-25e2-4a13-9476-f731e526e11b" (UID: "7ca8f66d-25e2-4a13-9476-f731e526e11b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:16.815120 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.815088    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-console-config" (OuterVolumeSpecName: "console-config") pod "7ca8f66d-25e2-4a13-9476-f731e526e11b" (UID: "7ca8f66d-25e2-4a13-9476-f731e526e11b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:16.816635 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.816600    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ca8f66d-25e2-4a13-9476-f731e526e11b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7ca8f66d-25e2-4a13-9476-f731e526e11b" (UID: "7ca8f66d-25e2-4a13-9476-f731e526e11b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:16.816743 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.816652    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ca8f66d-25e2-4a13-9476-f731e526e11b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7ca8f66d-25e2-4a13-9476-f731e526e11b" (UID: "7ca8f66d-25e2-4a13-9476-f731e526e11b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:16.816743 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.816667    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca8f66d-25e2-4a13-9476-f731e526e11b-kube-api-access-fbk4v" (OuterVolumeSpecName: "kube-api-access-fbk4v") pod "7ca8f66d-25e2-4a13-9476-f731e526e11b" (UID: "7ca8f66d-25e2-4a13-9476-f731e526e11b"). InnerVolumeSpecName "kube-api-access-fbk4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:16.915812 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.915775    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-oauth-serving-cert\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:47:16.915812 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.915808    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ca8f66d-25e2-4a13-9476-f731e526e11b-console-config\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:47:16.915812 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.915819    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ca8f66d-25e2-4a13-9476-f731e526e11b-console-serving-cert\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:47:16.916047 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.915827    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ca8f66d-25e2-4a13-9476-f731e526e11b-console-oauth-config\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:47:16.916047 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:16.915836    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbk4v\" (UniqueName: \"kubernetes.io/projected/7ca8f66d-25e2-4a13-9476-f731e526e11b-kube-api-access-fbk4v\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:47:17.432926 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:17.432885    2565 generic.go:356] "Generic (PLEG): container finished" podID="7ca8f66d-25e2-4a13-9476-f731e526e11b" containerID="4032062f14ae2f245fb43cb088f80251defa383bba56ca1caa6966225453a031" exitCode=0
Sep 16 18:47:17.433137 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:17.432950    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbb95445f-hsgwv"
Sep 16 18:47:17.433137 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:17.432981    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbb95445f-hsgwv" event={"ID":"7ca8f66d-25e2-4a13-9476-f731e526e11b","Type":"ContainerDied","Data":"4032062f14ae2f245fb43cb088f80251defa383bba56ca1caa6966225453a031"}
Sep 16 18:47:17.433137 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:17.433045    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbb95445f-hsgwv" event={"ID":"7ca8f66d-25e2-4a13-9476-f731e526e11b","Type":"ContainerDied","Data":"e5c23e1070823eb5e4a1a335d347eb4db27ed54462315aed44864d873988a5ec"}
Sep 16 18:47:17.433137 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:17.433072    2565 scope.go:117] "RemoveContainer" containerID="4032062f14ae2f245fb43cb088f80251defa383bba56ca1caa6966225453a031"
Sep 16 18:47:17.443650 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:17.443627    2565 scope.go:117] "RemoveContainer" containerID="4032062f14ae2f245fb43cb088f80251defa383bba56ca1caa6966225453a031"
Sep 16 18:47:17.443963 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:47:17.443946    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4032062f14ae2f245fb43cb088f80251defa383bba56ca1caa6966225453a031\": container with ID starting with 4032062f14ae2f245fb43cb088f80251defa383bba56ca1caa6966225453a031 not found: ID does not exist" containerID="4032062f14ae2f245fb43cb088f80251defa383bba56ca1caa6966225453a031"
Sep 16 18:47:17.444017 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:17.443971    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4032062f14ae2f245fb43cb088f80251defa383bba56ca1caa6966225453a031"} err="failed to get container status \"4032062f14ae2f245fb43cb088f80251defa383bba56ca1caa6966225453a031\": rpc error: code = NotFound desc = could not find container \"4032062f14ae2f245fb43cb088f80251defa383bba56ca1caa6966225453a031\": container with ID starting with 4032062f14ae2f245fb43cb088f80251defa383bba56ca1caa6966225453a031 not found: ID does not exist"
Sep 16 18:47:17.457260 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:17.457227    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fbb95445f-hsgwv"]
Sep 16 18:47:17.461766 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:17.461737    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fbb95445f-hsgwv"]
Sep 16 18:47:18.207359 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:18.207317    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ca8f66d-25e2-4a13-9476-f731e526e11b" path="/var/lib/kubelet/pods/7ca8f66d-25e2-4a13-9476-f731e526e11b/volumes"
Sep 16 18:47:36.628684 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:36.628647    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sgh8r"]
Sep 16 18:47:36.629361 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:36.629049    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ca8f66d-25e2-4a13-9476-f731e526e11b" containerName="console"
Sep 16 18:47:36.629361 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:36.629062    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ca8f66d-25e2-4a13-9476-f731e526e11b" containerName="console"
Sep 16 18:47:36.629361 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:36.629131    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ca8f66d-25e2-4a13-9476-f731e526e11b" containerName="console"
Sep 16 18:47:36.635829 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:36.635798    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-sgh8r"
Sep 16 18:47:36.638581 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:36.638537    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 18:47:36.638754 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:36.638649    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-48428\""
Sep 16 18:47:36.644583 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:36.644554    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sgh8r"]
Sep 16 18:47:36.726889 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:36.726848    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sgh8r"]
Sep 16 18:47:36.791378 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:36.791304    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7cbae579-aafe-4c47-8a08-d975575bc50e-config-file\") pod \"limitador-limitador-8588749977-sgh8r\" (UID: \"7cbae579-aafe-4c47-8a08-d975575bc50e\") " pod="kuadrant-system/limitador-limitador-8588749977-sgh8r"
Sep 16 18:47:36.791581 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:36.791455    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5z2c\" (UniqueName: \"kubernetes.io/projected/7cbae579-aafe-4c47-8a08-d975575bc50e-kube-api-access-f5z2c\") pod \"limitador-limitador-8588749977-sgh8r\" (UID: \"7cbae579-aafe-4c47-8a08-d975575bc50e\") " pod="kuadrant-system/limitador-limitador-8588749977-sgh8r"
Sep 16 18:47:36.892682 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:36.892586    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5z2c\" (UniqueName: \"kubernetes.io/projected/7cbae579-aafe-4c47-8a08-d975575bc50e-kube-api-access-f5z2c\") pod \"limitador-limitador-8588749977-sgh8r\" (UID: \"7cbae579-aafe-4c47-8a08-d975575bc50e\") " pod="kuadrant-system/limitador-limitador-8588749977-sgh8r"
Sep 16 18:47:36.892682 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:36.892657    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7cbae579-aafe-4c47-8a08-d975575bc50e-config-file\") pod \"limitador-limitador-8588749977-sgh8r\" (UID: \"7cbae579-aafe-4c47-8a08-d975575bc50e\") " pod="kuadrant-system/limitador-limitador-8588749977-sgh8r"
Sep 16 18:47:36.893251 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:36.893232    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7cbae579-aafe-4c47-8a08-d975575bc50e-config-file\") pod \"limitador-limitador-8588749977-sgh8r\" (UID: \"7cbae579-aafe-4c47-8a08-d975575bc50e\") " pod="kuadrant-system/limitador-limitador-8588749977-sgh8r"
Sep 16 18:47:36.904595 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:36.904564    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5z2c\" (UniqueName: \"kubernetes.io/projected/7cbae579-aafe-4c47-8a08-d975575bc50e-kube-api-access-f5z2c\") pod \"limitador-limitador-8588749977-sgh8r\" (UID: \"7cbae579-aafe-4c47-8a08-d975575bc50e\") " pod="kuadrant-system/limitador-limitador-8588749977-sgh8r"
Sep 16 18:47:36.948130 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:36.948088    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-sgh8r"
Sep 16 18:47:37.228549 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:37.228519    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sgh8r"]
Sep 16 18:47:37.230813 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:47:37.230784    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cbae579_aafe_4c47_8a08_d975575bc50e.slice/crio-e4878f58c43710f1d5eb43f82f8f2121ac7d4a44b698cc7d3c439f55d3549618 WatchSource:0}: Error finding container e4878f58c43710f1d5eb43f82f8f2121ac7d4a44b698cc7d3c439f55d3549618: Status 404 returned error can't find the container with id e4878f58c43710f1d5eb43f82f8f2121ac7d4a44b698cc7d3c439f55d3549618
Sep 16 18:47:37.544162 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:37.544065    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-sgh8r" event={"ID":"7cbae579-aafe-4c47-8a08-d975575bc50e","Type":"ContainerStarted","Data":"e4878f58c43710f1d5eb43f82f8f2121ac7d4a44b698cc7d3c439f55d3549618"}
Sep 16 18:47:38.551088 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:38.551045    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-sgh8r" event={"ID":"7cbae579-aafe-4c47-8a08-d975575bc50e","Type":"ContainerStarted","Data":"239bcb7aaff2fde26e4db2d331c371f51406df106ae75fb30e616f370ddbe63c"}
Sep 16 18:47:38.551532 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:38.551111    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-sgh8r"
Sep 16 18:47:38.576881 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:38.576819    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-sgh8r" podStartSLOduration=1.439382292 podStartE2EDuration="2.576801384s" totalImagesPullingTime="1.137419092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:38.575910934 +0000 UTC m=+496.935428920" watchObservedRunningTime="2026-09-16 18:47:38.576801384 +0000 UTC m=+496.936319379"
Sep 16 18:47:49.557094 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:49.557064    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-sgh8r"
Sep 16 18:47:49.917773 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:49.917680    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sgh8r"]
Sep 16 18:47:49.917963 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:49.917886    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-sgh8r" podUID="7cbae579-aafe-4c47-8a08-d975575bc50e" containerName="limitador" containerID="cri-o://239bcb7aaff2fde26e4db2d331c371f51406df106ae75fb30e616f370ddbe63c" gracePeriod=30
Sep 16 18:47:50.468110 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:50.468078    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-sgh8r"
Sep 16 18:47:50.500282 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:50.500199    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5z2c\" (UniqueName: \"kubernetes.io/projected/7cbae579-aafe-4c47-8a08-d975575bc50e-kube-api-access-f5z2c\") pod \"7cbae579-aafe-4c47-8a08-d975575bc50e\" (UID: \"7cbae579-aafe-4c47-8a08-d975575bc50e\") "
Sep 16 18:47:50.500454 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:50.500403    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7cbae579-aafe-4c47-8a08-d975575bc50e-config-file\") pod \"7cbae579-aafe-4c47-8a08-d975575bc50e\" (UID: \"7cbae579-aafe-4c47-8a08-d975575bc50e\") "
Sep 16 18:47:50.500752 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:50.500720    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cbae579-aafe-4c47-8a08-d975575bc50e-config-file" (OuterVolumeSpecName: "config-file") pod "7cbae579-aafe-4c47-8a08-d975575bc50e" (UID: "7cbae579-aafe-4c47-8a08-d975575bc50e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:50.502440 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:50.502417    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cbae579-aafe-4c47-8a08-d975575bc50e-kube-api-access-f5z2c" (OuterVolumeSpecName: "kube-api-access-f5z2c") pod "7cbae579-aafe-4c47-8a08-d975575bc50e" (UID: "7cbae579-aafe-4c47-8a08-d975575bc50e"). InnerVolumeSpecName "kube-api-access-f5z2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:50.601356 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:50.601306    2565 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7cbae579-aafe-4c47-8a08-d975575bc50e-config-file\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:47:50.601356 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:50.601339    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5z2c\" (UniqueName: \"kubernetes.io/projected/7cbae579-aafe-4c47-8a08-d975575bc50e-kube-api-access-f5z2c\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:47:50.620173 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:50.620139    2565 generic.go:356] "Generic (PLEG): container finished" podID="7cbae579-aafe-4c47-8a08-d975575bc50e" containerID="239bcb7aaff2fde26e4db2d331c371f51406df106ae75fb30e616f370ddbe63c" exitCode=0
Sep 16 18:47:50.620336 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:50.620184    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-sgh8r"
Sep 16 18:47:50.620336 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:50.620218    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-sgh8r" event={"ID":"7cbae579-aafe-4c47-8a08-d975575bc50e","Type":"ContainerDied","Data":"239bcb7aaff2fde26e4db2d331c371f51406df106ae75fb30e616f370ddbe63c"}
Sep 16 18:47:50.620336 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:50.620270    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-sgh8r" event={"ID":"7cbae579-aafe-4c47-8a08-d975575bc50e","Type":"ContainerDied","Data":"e4878f58c43710f1d5eb43f82f8f2121ac7d4a44b698cc7d3c439f55d3549618"}
Sep 16 18:47:50.620336 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:50.620292    2565 scope.go:117] "RemoveContainer" containerID="239bcb7aaff2fde26e4db2d331c371f51406df106ae75fb30e616f370ddbe63c"
Sep 16 18:47:50.630147 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:50.630124    2565 scope.go:117] "RemoveContainer" containerID="239bcb7aaff2fde26e4db2d331c371f51406df106ae75fb30e616f370ddbe63c"
Sep 16 18:47:50.630505 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:47:50.630485    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"239bcb7aaff2fde26e4db2d331c371f51406df106ae75fb30e616f370ddbe63c\": container with ID starting with 239bcb7aaff2fde26e4db2d331c371f51406df106ae75fb30e616f370ddbe63c not found: ID does not exist" containerID="239bcb7aaff2fde26e4db2d331c371f51406df106ae75fb30e616f370ddbe63c"
Sep 16 18:47:50.630578 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:50.630517    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"239bcb7aaff2fde26e4db2d331c371f51406df106ae75fb30e616f370ddbe63c"} err="failed to get container status \"239bcb7aaff2fde26e4db2d331c371f51406df106ae75fb30e616f370ddbe63c\": rpc error: code = NotFound desc = could not find container \"239bcb7aaff2fde26e4db2d331c371f51406df106ae75fb30e616f370ddbe63c\": container with ID starting with 239bcb7aaff2fde26e4db2d331c371f51406df106ae75fb30e616f370ddbe63c not found: ID does not exist"
Sep 16 18:47:50.646556 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:50.646522    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sgh8r"]
Sep 16 18:47:50.653558 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:50.653527    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sgh8r"]
Sep 16 18:47:52.207709 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:47:52.207674    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cbae579-aafe-4c47-8a08-d975575bc50e" path="/var/lib/kubelet/pods/7cbae579-aafe-4c47-8a08-d975575bc50e/volumes"
Sep 16 18:48:19.678841 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.678752    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"]
Sep 16 18:48:19.679255 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.679116    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cbae579-aafe-4c47-8a08-d975575bc50e" containerName="limitador"
Sep 16 18:48:19.679255 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.679128    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cbae579-aafe-4c47-8a08-d975575bc50e" containerName="limitador"
Sep 16 18:48:19.679255 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.679193    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cbae579-aafe-4c47-8a08-d975575bc50e" containerName="limitador"
Sep 16 18:48:19.682340 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.682315    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.701565 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.701533    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"]
Sep 16 18:48:19.750572 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.750534    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2adc2cd-afaf-4150-8963-7d2b524b47e4-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.750760 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.750592    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2adc2cd-afaf-4150-8963-7d2b524b47e4-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.750760 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.750613    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e2adc2cd-afaf-4150-8963-7d2b524b47e4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.750760 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.750655    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e2adc2cd-afaf-4150-8963-7d2b524b47e4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.750760 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.750685    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2adc2cd-afaf-4150-8963-7d2b524b47e4-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.750760 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.750723    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2txw9\" (UniqueName: \"kubernetes.io/projected/e2adc2cd-afaf-4150-8963-7d2b524b47e4-kube-api-access-2txw9\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.750921 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.750774    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e2adc2cd-afaf-4150-8963-7d2b524b47e4-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.851588 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.851544    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2adc2cd-afaf-4150-8963-7d2b524b47e4-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.851588 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.851589    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2txw9\" (UniqueName: \"kubernetes.io/projected/e2adc2cd-afaf-4150-8963-7d2b524b47e4-kube-api-access-2txw9\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.851838 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.851636    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e2adc2cd-afaf-4150-8963-7d2b524b47e4-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.851838 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.851668    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2adc2cd-afaf-4150-8963-7d2b524b47e4-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.851838 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.851716    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2adc2cd-afaf-4150-8963-7d2b524b47e4-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.851838 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.851743    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e2adc2cd-afaf-4150-8963-7d2b524b47e4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.851838 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.851773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e2adc2cd-afaf-4150-8963-7d2b524b47e4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.852652 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.852620    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e2adc2cd-afaf-4150-8963-7d2b524b47e4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.854152 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.854129    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2adc2cd-afaf-4150-8963-7d2b524b47e4-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.854370 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.854330    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e2adc2cd-afaf-4150-8963-7d2b524b47e4-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.854447 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.854366    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2adc2cd-afaf-4150-8963-7d2b524b47e4-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.854485 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.854466    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e2adc2cd-afaf-4150-8963-7d2b524b47e4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.865231 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.865203    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2adc2cd-afaf-4150-8963-7d2b524b47e4-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.866330 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.866306    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2txw9\" (UniqueName: \"kubernetes.io/projected/e2adc2cd-afaf-4150-8963-7d2b524b47e4-kube-api-access-2txw9\") pod \"istiod-openshift-gateway-5677ccb49b-qg4vd\" (UID: \"e2adc2cd-afaf-4150-8963-7d2b524b47e4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:19.991762 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:19.991723    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:20.275791 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:20.275764    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"]
Sep 16 18:48:20.277745 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:48:20.277719    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2adc2cd_afaf_4150_8963_7d2b524b47e4.slice/crio-7bc7c2acc12dd8c1e82a9bfd8d269791201d797819d597ffd80ee02d4f988d35 WatchSource:0}: Error finding container 7bc7c2acc12dd8c1e82a9bfd8d269791201d797819d597ffd80ee02d4f988d35: Status 404 returned error can't find the container with id 7bc7c2acc12dd8c1e82a9bfd8d269791201d797819d597ffd80ee02d4f988d35
Sep 16 18:48:20.279860 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:20.279825    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:48:20.279962 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:20.279904    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:48:20.775489 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:20.775437    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd" event={"ID":"e2adc2cd-afaf-4150-8963-7d2b524b47e4","Type":"ContainerStarted","Data":"265082fcaf12ff4cfd18ee830de49a1c929077fde3fa2fe93190221126fc4b40"}
Sep 16 18:48:20.776055 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:20.775507    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd" event={"ID":"e2adc2cd-afaf-4150-8963-7d2b524b47e4","Type":"ContainerStarted","Data":"7bc7c2acc12dd8c1e82a9bfd8d269791201d797819d597ffd80ee02d4f988d35"}
Sep 16 18:48:20.776055 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:20.775610    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:20.800712 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:20.800646    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd" podStartSLOduration=1.8006263599999999 podStartE2EDuration="1.80062636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:20.797097602 +0000 UTC m=+539.156615599" watchObservedRunningTime="2026-09-16 18:48:20.80062636 +0000 UTC m=+539.160144355"
Sep 16 18:48:21.782852 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:21.782823    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qg4vd"
Sep 16 18:48:21.875783 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:21.875743    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"]
Sep 16 18:48:21.876016 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:21.875988    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb" podUID="8a740f62-b581-4c0a-8cda-58cb1431e843" containerName="discovery" containerID="cri-o://ba258de62594b371c9ee4cb872531e8ac54b1f786f18fb2e6cc2131854790197" gracePeriod=30
Sep 16 18:48:22.151285 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.151257    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:48:22.273122 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.273083    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtfjr\" (UniqueName: \"kubernetes.io/projected/8a740f62-b581-4c0a-8cda-58cb1431e843-kube-api-access-wtfjr\") pod \"8a740f62-b581-4c0a-8cda-58cb1431e843\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") "
Sep 16 18:48:22.273122 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.273125    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-csr-ca-configmap\") pod \"8a740f62-b581-4c0a-8cda-58cb1431e843\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") "
Sep 16 18:48:22.273417 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.273158    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8a740f62-b581-4c0a-8cda-58cb1431e843-cacerts\") pod \"8a740f62-b581-4c0a-8cda-58cb1431e843\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") "
Sep 16 18:48:22.273417 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.273189    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-csr-dns-cert\") pod \"8a740f62-b581-4c0a-8cda-58cb1431e843\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") "
Sep 16 18:48:22.273417 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.273249    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-kubeconfig\") pod \"8a740f62-b581-4c0a-8cda-58cb1431e843\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") "
Sep 16 18:48:22.273417 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.273303    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-token\") pod \"8a740f62-b581-4c0a-8cda-58cb1431e843\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") "
Sep 16 18:48:22.273417 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.273330    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a740f62-b581-4c0a-8cda-58cb1431e843-local-certs\") pod \"8a740f62-b581-4c0a-8cda-58cb1431e843\" (UID: \"8a740f62-b581-4c0a-8cda-58cb1431e843\") "
Sep 16 18:48:22.273666 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.273616    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "8a740f62-b581-4c0a-8cda-58cb1431e843" (UID: "8a740f62-b581-4c0a-8cda-58cb1431e843"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:22.275804 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.275761    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a740f62-b581-4c0a-8cda-58cb1431e843-kube-api-access-wtfjr" (OuterVolumeSpecName: "kube-api-access-wtfjr") pod "8a740f62-b581-4c0a-8cda-58cb1431e843" (UID: "8a740f62-b581-4c0a-8cda-58cb1431e843"). InnerVolumeSpecName "kube-api-access-wtfjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:22.275804 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.275783    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-token" (OuterVolumeSpecName: "istio-token") pod "8a740f62-b581-4c0a-8cda-58cb1431e843" (UID: "8a740f62-b581-4c0a-8cda-58cb1431e843"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:22.276054 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.276019    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "8a740f62-b581-4c0a-8cda-58cb1431e843" (UID: "8a740f62-b581-4c0a-8cda-58cb1431e843"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:22.276160 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.276062    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a740f62-b581-4c0a-8cda-58cb1431e843-cacerts" (OuterVolumeSpecName: "cacerts") pod "8a740f62-b581-4c0a-8cda-58cb1431e843" (UID: "8a740f62-b581-4c0a-8cda-58cb1431e843"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:22.276160 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.276083    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "8a740f62-b581-4c0a-8cda-58cb1431e843" (UID: "8a740f62-b581-4c0a-8cda-58cb1431e843"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:22.276160 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.276093    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a740f62-b581-4c0a-8cda-58cb1431e843-local-certs" (OuterVolumeSpecName: "local-certs") pod "8a740f62-b581-4c0a-8cda-58cb1431e843" (UID: "8a740f62-b581-4c0a-8cda-58cb1431e843"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:22.374585 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.374499    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtfjr\" (UniqueName: \"kubernetes.io/projected/8a740f62-b581-4c0a-8cda-58cb1431e843-kube-api-access-wtfjr\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:48:22.374585 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.374526    2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-csr-ca-configmap\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:48:22.374585 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.374536    2565 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8a740f62-b581-4c0a-8cda-58cb1431e843-cacerts\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:48:22.374585 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.374546    2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-csr-dns-cert\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:48:22.374585 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.374556    2565 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-kubeconfig\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:48:22.374585 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.374564    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a740f62-b581-4c0a-8cda-58cb1431e843-istio-token\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:48:22.374585 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.374572    2565 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a740f62-b581-4c0a-8cda-58cb1431e843-local-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:48:22.786673 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.786637    2565 generic.go:356] "Generic (PLEG): container finished" podID="8a740f62-b581-4c0a-8cda-58cb1431e843" containerID="ba258de62594b371c9ee4cb872531e8ac54b1f786f18fb2e6cc2131854790197" exitCode=0
Sep 16 18:48:22.787092 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.786687    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"
Sep 16 18:48:22.787092 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.786720    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb" event={"ID":"8a740f62-b581-4c0a-8cda-58cb1431e843","Type":"ContainerDied","Data":"ba258de62594b371c9ee4cb872531e8ac54b1f786f18fb2e6cc2131854790197"}
Sep 16 18:48:22.787092 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.786769    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb" event={"ID":"8a740f62-b581-4c0a-8cda-58cb1431e843","Type":"ContainerDied","Data":"69c08e2718e986086eeaab6471f2efd64495ab95898017512552adccef0c5f61"}
Sep 16 18:48:22.787092 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.786791    2565 scope.go:117] "RemoveContainer" containerID="ba258de62594b371c9ee4cb872531e8ac54b1f786f18fb2e6cc2131854790197"
Sep 16 18:48:22.795955 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.795934    2565 scope.go:117] "RemoveContainer" containerID="ba258de62594b371c9ee4cb872531e8ac54b1f786f18fb2e6cc2131854790197"
Sep 16 18:48:22.796258 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:48:22.796240    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba258de62594b371c9ee4cb872531e8ac54b1f786f18fb2e6cc2131854790197\": container with ID starting with ba258de62594b371c9ee4cb872531e8ac54b1f786f18fb2e6cc2131854790197 not found: ID does not exist" containerID="ba258de62594b371c9ee4cb872531e8ac54b1f786f18fb2e6cc2131854790197"
Sep 16 18:48:22.796337 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.796268    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba258de62594b371c9ee4cb872531e8ac54b1f786f18fb2e6cc2131854790197"} err="failed to get container status \"ba258de62594b371c9ee4cb872531e8ac54b1f786f18fb2e6cc2131854790197\": rpc error: code = NotFound desc = could not find container \"ba258de62594b371c9ee4cb872531e8ac54b1f786f18fb2e6cc2131854790197\": container with ID starting with ba258de62594b371c9ee4cb872531e8ac54b1f786f18fb2e6cc2131854790197 not found: ID does not exist"
Sep 16 18:48:22.810152 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.810124    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"]
Sep 16 18:48:22.816322 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:22.816294    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb"]
Sep 16 18:48:24.206507 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:24.206471    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a740f62-b581-4c0a-8cda-58cb1431e843" path="/var/lib/kubelet/pods/8a740f62-b581-4c0a-8cda-58cb1431e843/volumes"
Sep 16 18:48:27.092620 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:27.092572    2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-9bnkb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 18:48:27.093014 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:27.092652    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bnkb" podUID="8a740f62-b581-4c0a-8cda-58cb1431e843" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.30:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 18:48:33.975515 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:33.975473    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-557b59fdc4-l8zmh"]
Sep 16 18:48:33.975921 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:33.975898    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a740f62-b581-4c0a-8cda-58cb1431e843" containerName="discovery"
Sep 16 18:48:33.975921 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:33.975910    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a740f62-b581-4c0a-8cda-58cb1431e843" containerName="discovery"
Sep 16 18:48:33.976001 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:33.975968    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a740f62-b581-4c0a-8cda-58cb1431e843" containerName="discovery"
Sep 16 18:48:33.980569 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:33.980544    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-557b59fdc4-l8zmh"
Sep 16 18:48:33.983019 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:33.982994    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 16 18:48:33.983019 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:33.983004    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:48:33.983727 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:33.983708    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-zsdkm\""
Sep 16 18:48:33.983803 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:33.983709    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:48:33.993714 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:33.993655    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-557b59fdc4-l8zmh"]
Sep 16 18:48:33.996588 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:33.996561    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6f9695cb4c-zvctb"]
Sep 16 18:48:34.000402 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.000379    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6f9695cb4c-zvctb"
Sep 16 18:48:34.003327 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.003297    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 16 18:48:34.003666 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.003645    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-ldrhc\""
Sep 16 18:48:34.011664 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.011634    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6f9695cb4c-zvctb"]
Sep 16 18:48:34.017789 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.017760    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-d825x"]
Sep 16 18:48:34.021194 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.021171    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-d825x"
Sep 16 18:48:34.023579 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.023551    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-rqbgn\""
Sep 16 18:48:34.023914 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.023871    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:48:34.032013 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.031984    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-d825x"]
Sep 16 18:48:34.071032 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.070996    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4mns\" (UniqueName: \"kubernetes.io/projected/3800552f-1424-421a-aa4a-4a2c5f60cc98-kube-api-access-h4mns\") pod \"kserve-controller-manager-557b59fdc4-l8zmh\" (UID: \"3800552f-1424-421a-aa4a-4a2c5f60cc98\") " pod="kserve/kserve-controller-manager-557b59fdc4-l8zmh"
Sep 16 18:48:34.071208 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.071041    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mx72\" (UniqueName: \"kubernetes.io/projected/c7f28c0c-962c-4b18-ad31-9fb9605873f5-kube-api-access-8mx72\") pod \"llmisvc-controller-manager-6f9695cb4c-zvctb\" (UID: \"c7f28c0c-962c-4b18-ad31-9fb9605873f5\") " pod="kserve/llmisvc-controller-manager-6f9695cb4c-zvctb"
Sep 16 18:48:34.071208 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.071135    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3800552f-1424-421a-aa4a-4a2c5f60cc98-cert\") pod \"kserve-controller-manager-557b59fdc4-l8zmh\" (UID: \"3800552f-1424-421a-aa4a-4a2c5f60cc98\") " pod="kserve/kserve-controller-manager-557b59fdc4-l8zmh"
Sep 16 18:48:34.071208 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.071189    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7f28c0c-962c-4b18-ad31-9fb9605873f5-cert\") pod \"llmisvc-controller-manager-6f9695cb4c-zvctb\" (UID: \"c7f28c0c-962c-4b18-ad31-9fb9605873f5\") " pod="kserve/llmisvc-controller-manager-6f9695cb4c-zvctb"
Sep 16 18:48:34.171789 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.171746    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/247f31d1-7a74-415f-8fdb-ea5289f128ac-data\") pod \"seaweedfs-64568bcd49-d825x\" (UID: \"247f31d1-7a74-415f-8fdb-ea5289f128ac\") " pod="kserve/seaweedfs-64568bcd49-d825x"
Sep 16 18:48:34.171789 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.171792    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7f28c0c-962c-4b18-ad31-9fb9605873f5-cert\") pod \"llmisvc-controller-manager-6f9695cb4c-zvctb\" (UID: \"c7f28c0c-962c-4b18-ad31-9fb9605873f5\") " pod="kserve/llmisvc-controller-manager-6f9695cb4c-zvctb"
Sep 16 18:48:34.172027 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.171813    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4xkf\" (UniqueName: \"kubernetes.io/projected/247f31d1-7a74-415f-8fdb-ea5289f128ac-kube-api-access-l4xkf\") pod \"seaweedfs-64568bcd49-d825x\" (UID: \"247f31d1-7a74-415f-8fdb-ea5289f128ac\") " pod="kserve/seaweedfs-64568bcd49-d825x"
Sep 16 18:48:34.172027 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.171898    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4mns\" (UniqueName: \"kubernetes.io/projected/3800552f-1424-421a-aa4a-4a2c5f60cc98-kube-api-access-h4mns\") pod \"kserve-controller-manager-557b59fdc4-l8zmh\" (UID: \"3800552f-1424-421a-aa4a-4a2c5f60cc98\") " pod="kserve/kserve-controller-manager-557b59fdc4-l8zmh"
Sep 16 18:48:34.172027 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:48:34.171930    2565 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 16 18:48:34.172027 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.171951    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mx72\" (UniqueName: \"kubernetes.io/projected/c7f28c0c-962c-4b18-ad31-9fb9605873f5-kube-api-access-8mx72\") pod \"llmisvc-controller-manager-6f9695cb4c-zvctb\" (UID: \"c7f28c0c-962c-4b18-ad31-9fb9605873f5\") " pod="kserve/llmisvc-controller-manager-6f9695cb4c-zvctb"
Sep 16 18:48:34.172027 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:48:34.172009    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f28c0c-962c-4b18-ad31-9fb9605873f5-cert podName:c7f28c0c-962c-4b18-ad31-9fb9605873f5 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:34.67198837 +0000 UTC m=+553.031506357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c7f28c0c-962c-4b18-ad31-9fb9605873f5-cert") pod "llmisvc-controller-manager-6f9695cb4c-zvctb" (UID: "c7f28c0c-962c-4b18-ad31-9fb9605873f5") : secret "llmisvc-webhook-server-cert" not found
Sep 16 18:48:34.172256 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.172052    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3800552f-1424-421a-aa4a-4a2c5f60cc98-cert\") pod \"kserve-controller-manager-557b59fdc4-l8zmh\" (UID: \"3800552f-1424-421a-aa4a-4a2c5f60cc98\") " pod="kserve/kserve-controller-manager-557b59fdc4-l8zmh"
Sep 16 18:48:34.174598 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.174569    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3800552f-1424-421a-aa4a-4a2c5f60cc98-cert\") pod \"kserve-controller-manager-557b59fdc4-l8zmh\" (UID: \"3800552f-1424-421a-aa4a-4a2c5f60cc98\") " pod="kserve/kserve-controller-manager-557b59fdc4-l8zmh"
Sep 16 18:48:34.182913 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.182890    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mx72\" (UniqueName: \"kubernetes.io/projected/c7f28c0c-962c-4b18-ad31-9fb9605873f5-kube-api-access-8mx72\") pod \"llmisvc-controller-manager-6f9695cb4c-zvctb\" (UID: \"c7f28c0c-962c-4b18-ad31-9fb9605873f5\") " pod="kserve/llmisvc-controller-manager-6f9695cb4c-zvctb"
Sep 16 18:48:34.183106 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.183082    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4mns\" (UniqueName: \"kubernetes.io/projected/3800552f-1424-421a-aa4a-4a2c5f60cc98-kube-api-access-h4mns\") pod \"kserve-controller-manager-557b59fdc4-l8zmh\" (UID: \"3800552f-1424-421a-aa4a-4a2c5f60cc98\") " pod="kserve/kserve-controller-manager-557b59fdc4-l8zmh"
Sep 16 18:48:34.272804 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.272693    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/247f31d1-7a74-415f-8fdb-ea5289f128ac-data\") pod \"seaweedfs-64568bcd49-d825x\" (UID: \"247f31d1-7a74-415f-8fdb-ea5289f128ac\") " pod="kserve/seaweedfs-64568bcd49-d825x"
Sep 16 18:48:34.272804 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.272760    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4xkf\" (UniqueName: \"kubernetes.io/projected/247f31d1-7a74-415f-8fdb-ea5289f128ac-kube-api-access-l4xkf\") pod \"seaweedfs-64568bcd49-d825x\" (UID: \"247f31d1-7a74-415f-8fdb-ea5289f128ac\") " pod="kserve/seaweedfs-64568bcd49-d825x"
Sep 16 18:48:34.273184 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.273162    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/247f31d1-7a74-415f-8fdb-ea5289f128ac-data\") pod \"seaweedfs-64568bcd49-d825x\" (UID: \"247f31d1-7a74-415f-8fdb-ea5289f128ac\") " pod="kserve/seaweedfs-64568bcd49-d825x"
Sep 16 18:48:34.282736 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.282701    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4xkf\" (UniqueName: \"kubernetes.io/projected/247f31d1-7a74-415f-8fdb-ea5289f128ac-kube-api-access-l4xkf\") pod \"seaweedfs-64568bcd49-d825x\" (UID: \"247f31d1-7a74-415f-8fdb-ea5289f128ac\") " pod="kserve/seaweedfs-64568bcd49-d825x"
Sep 16 18:48:34.292562 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.292534    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-557b59fdc4-l8zmh"
Sep 16 18:48:34.333015 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.332973    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-d825x"
Sep 16 18:48:34.577517 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.577487    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-557b59fdc4-l8zmh"]
Sep 16 18:48:34.579223 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:48:34.579195    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3800552f_1424_421a_aa4a_4a2c5f60cc98.slice/crio-d4e44fb9a57e84af54e7229347bebbdc3b304f9892fa6812a81e47778deb666e WatchSource:0}: Error finding container d4e44fb9a57e84af54e7229347bebbdc3b304f9892fa6812a81e47778deb666e: Status 404 returned error can't find the container with id d4e44fb9a57e84af54e7229347bebbdc3b304f9892fa6812a81e47778deb666e
Sep 16 18:48:34.619219 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.619191    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-d825x"]
Sep 16 18:48:34.620928 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:48:34.620904    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247f31d1_7a74_415f_8fdb_ea5289f128ac.slice/crio-f74f68c06e1a4634ad094e16bd1adb8ff9bad42bb02aaeab9cd364bdd12948db WatchSource:0}: Error finding container f74f68c06e1a4634ad094e16bd1adb8ff9bad42bb02aaeab9cd364bdd12948db: Status 404 returned error can't find the container with id f74f68c06e1a4634ad094e16bd1adb8ff9bad42bb02aaeab9cd364bdd12948db
Sep 16 18:48:34.675805 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.675763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7f28c0c-962c-4b18-ad31-9fb9605873f5-cert\") pod \"llmisvc-controller-manager-6f9695cb4c-zvctb\" (UID: \"c7f28c0c-962c-4b18-ad31-9fb9605873f5\") " pod="kserve/llmisvc-controller-manager-6f9695cb4c-zvctb"
Sep 16 18:48:34.675974 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:48:34.675912    2565 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 16 18:48:34.676021 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:48:34.675981    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f28c0c-962c-4b18-ad31-9fb9605873f5-cert podName:c7f28c0c-962c-4b18-ad31-9fb9605873f5 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:35.675963998 +0000 UTC m=+554.035481973 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c7f28c0c-962c-4b18-ad31-9fb9605873f5-cert") pod "llmisvc-controller-manager-6f9695cb4c-zvctb" (UID: "c7f28c0c-962c-4b18-ad31-9fb9605873f5") : secret "llmisvc-webhook-server-cert" not found
Sep 16 18:48:34.851929 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.851838    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-557b59fdc4-l8zmh" event={"ID":"3800552f-1424-421a-aa4a-4a2c5f60cc98","Type":"ContainerStarted","Data":"d4e44fb9a57e84af54e7229347bebbdc3b304f9892fa6812a81e47778deb666e"}
Sep 16 18:48:34.853166 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:34.853140    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-d825x" event={"ID":"247f31d1-7a74-415f-8fdb-ea5289f128ac","Type":"ContainerStarted","Data":"f74f68c06e1a4634ad094e16bd1adb8ff9bad42bb02aaeab9cd364bdd12948db"}
Sep 16 18:48:35.686800 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:35.686754    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7f28c0c-962c-4b18-ad31-9fb9605873f5-cert\") pod \"llmisvc-controller-manager-6f9695cb4c-zvctb\" (UID: \"c7f28c0c-962c-4b18-ad31-9fb9605873f5\") " pod="kserve/llmisvc-controller-manager-6f9695cb4c-zvctb"
Sep 16 18:48:35.690132 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:35.690100    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7f28c0c-962c-4b18-ad31-9fb9605873f5-cert\") pod \"llmisvc-controller-manager-6f9695cb4c-zvctb\" (UID: \"c7f28c0c-962c-4b18-ad31-9fb9605873f5\") " pod="kserve/llmisvc-controller-manager-6f9695cb4c-zvctb"
Sep 16 18:48:35.814337 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:35.814297    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6f9695cb4c-zvctb"
Sep 16 18:48:38.402489 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:38.402453    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6f9695cb4c-zvctb"]
Sep 16 18:48:38.404281 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:48:38.404249    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc7f28c0c_962c_4b18_ad31_9fb9605873f5.slice/crio-1067ca2ac98cc4bfbfb627809ef32643fdf5c50259b8d908dea80e455899f93f WatchSource:0}: Error finding container 1067ca2ac98cc4bfbfb627809ef32643fdf5c50259b8d908dea80e455899f93f: Status 404 returned error can't find the container with id 1067ca2ac98cc4bfbfb627809ef32643fdf5c50259b8d908dea80e455899f93f
Sep 16 18:48:38.881752 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:38.881708    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-d825x" event={"ID":"247f31d1-7a74-415f-8fdb-ea5289f128ac","Type":"ContainerStarted","Data":"33d10fcb6d642e0e8495f79cd2a5cd9b8da04ed08ab9b8c29f51bcac1bc2768c"}
Sep 16 18:48:38.881957 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:38.881797    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-d825x"
Sep 16 18:48:38.883324 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:38.883294    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-557b59fdc4-l8zmh" event={"ID":"3800552f-1424-421a-aa4a-4a2c5f60cc98","Type":"ContainerStarted","Data":"879b9a76dcaab9485107606b2d553426365d580eee022ffe748d104090f5ae5a"}
Sep 16 18:48:38.883474 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:38.883377    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-557b59fdc4-l8zmh"
Sep 16 18:48:38.884650 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:38.884619    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6f9695cb4c-zvctb" event={"ID":"c7f28c0c-962c-4b18-ad31-9fb9605873f5","Type":"ContainerStarted","Data":"1067ca2ac98cc4bfbfb627809ef32643fdf5c50259b8d908dea80e455899f93f"}
Sep 16 18:48:38.910049 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:38.909989    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-d825x" podStartSLOduration=1.253252257 podStartE2EDuration="4.909970704s" totalImagesPullingTime="3.656718447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:38.909102491 +0000 UTC m=+557.268620485" watchObservedRunningTime="2026-09-16 18:48:38.909970704 +0000 UTC m=+557.269488690"
Sep 16 18:48:38.937520 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:38.937465    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-557b59fdc4-l8zmh" podStartSLOduration=2.389081587 podStartE2EDuration="5.937445938s" totalImagesPullingTime="3.548364351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:38.935383939 +0000 UTC m=+557.294901934" watchObservedRunningTime="2026-09-16 18:48:38.937445938 +0000 UTC m=+557.296963934"
Sep 16 18:48:41.902769 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:41.902732    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6f9695cb4c-zvctb" event={"ID":"c7f28c0c-962c-4b18-ad31-9fb9605873f5","Type":"ContainerStarted","Data":"347c0dd889e9a95cfe0b3ca79697b6a019bbbfcf1422476022f42ae279720c85"}
Sep 16 18:48:41.903241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:41.902796    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6f9695cb4c-zvctb"
Sep 16 18:48:41.928209 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:41.928146    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6f9695cb4c-zvctb" podStartSLOduration=6.413170249 podStartE2EDuration="8.928128593s" totalImagesPullingTime="2.514958344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:41.92645812 +0000 UTC m=+560.285976125" watchObservedRunningTime="2026-09-16 18:48:41.928128593 +0000 UTC m=+560.287646587"
Sep 16 18:48:44.892463 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:48:44.892429    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-d825x"
Sep 16 18:49:09.896362 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:09.896314    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-557b59fdc4-l8zmh"
Sep 16 18:49:12.910901 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:12.910866    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6f9695cb4c-zvctb"
Sep 16 18:49:22.017922 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.017879    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-557b59fdc4-l8zmh"]
Sep 16 18:49:22.018361 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.018088    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-557b59fdc4-l8zmh" podUID="3800552f-1424-421a-aa4a-4a2c5f60cc98" containerName="manager" containerID="cri-o://879b9a76dcaab9485107606b2d553426365d580eee022ffe748d104090f5ae5a" gracePeriod=10
Sep 16 18:49:22.047607 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.047574    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-557b59fdc4-f6qxg"]
Sep 16 18:49:22.051108 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.051086    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-557b59fdc4-f6qxg"
Sep 16 18:49:22.062052 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.062012    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-557b59fdc4-f6qxg"]
Sep 16 18:49:22.205413 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.205382    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1126c57-2b37-4788-bb98-c043dde60b85-cert\") pod \"kserve-controller-manager-557b59fdc4-f6qxg\" (UID: \"c1126c57-2b37-4788-bb98-c043dde60b85\") " pod="kserve/kserve-controller-manager-557b59fdc4-f6qxg"
Sep 16 18:49:22.205601 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.205438    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdzp2\" (UniqueName: \"kubernetes.io/projected/c1126c57-2b37-4788-bb98-c043dde60b85-kube-api-access-bdzp2\") pod \"kserve-controller-manager-557b59fdc4-f6qxg\" (UID: \"c1126c57-2b37-4788-bb98-c043dde60b85\") " pod="kserve/kserve-controller-manager-557b59fdc4-f6qxg"
Sep 16 18:49:22.269945 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.269870    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-557b59fdc4-l8zmh"
Sep 16 18:49:22.306819 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.306776    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdzp2\" (UniqueName: \"kubernetes.io/projected/c1126c57-2b37-4788-bb98-c043dde60b85-kube-api-access-bdzp2\") pod \"kserve-controller-manager-557b59fdc4-f6qxg\" (UID: \"c1126c57-2b37-4788-bb98-c043dde60b85\") " pod="kserve/kserve-controller-manager-557b59fdc4-f6qxg"
Sep 16 18:49:22.307035 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.306915    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1126c57-2b37-4788-bb98-c043dde60b85-cert\") pod \"kserve-controller-manager-557b59fdc4-f6qxg\" (UID: \"c1126c57-2b37-4788-bb98-c043dde60b85\") " pod="kserve/kserve-controller-manager-557b59fdc4-f6qxg"
Sep 16 18:49:22.309510 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.309473    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1126c57-2b37-4788-bb98-c043dde60b85-cert\") pod \"kserve-controller-manager-557b59fdc4-f6qxg\" (UID: \"c1126c57-2b37-4788-bb98-c043dde60b85\") " pod="kserve/kserve-controller-manager-557b59fdc4-f6qxg"
Sep 16 18:49:22.315558 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.315522    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdzp2\" (UniqueName: \"kubernetes.io/projected/c1126c57-2b37-4788-bb98-c043dde60b85-kube-api-access-bdzp2\") pod \"kserve-controller-manager-557b59fdc4-f6qxg\" (UID: \"c1126c57-2b37-4788-bb98-c043dde60b85\") " pod="kserve/kserve-controller-manager-557b59fdc4-f6qxg"
Sep 16 18:49:22.406659 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.406620    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-557b59fdc4-f6qxg"
Sep 16 18:49:22.407760 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.407736    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3800552f-1424-421a-aa4a-4a2c5f60cc98-cert\") pod \"3800552f-1424-421a-aa4a-4a2c5f60cc98\" (UID: \"3800552f-1424-421a-aa4a-4a2c5f60cc98\") "
Sep 16 18:49:22.407828 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.407802    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4mns\" (UniqueName: \"kubernetes.io/projected/3800552f-1424-421a-aa4a-4a2c5f60cc98-kube-api-access-h4mns\") pod \"3800552f-1424-421a-aa4a-4a2c5f60cc98\" (UID: \"3800552f-1424-421a-aa4a-4a2c5f60cc98\") "
Sep 16 18:49:22.410141 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.410119    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3800552f-1424-421a-aa4a-4a2c5f60cc98-cert" (OuterVolumeSpecName: "cert") pod "3800552f-1424-421a-aa4a-4a2c5f60cc98" (UID: "3800552f-1424-421a-aa4a-4a2c5f60cc98"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:22.410277 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.410252    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3800552f-1424-421a-aa4a-4a2c5f60cc98-kube-api-access-h4mns" (OuterVolumeSpecName: "kube-api-access-h4mns") pod "3800552f-1424-421a-aa4a-4a2c5f60cc98" (UID: "3800552f-1424-421a-aa4a-4a2c5f60cc98"). InnerVolumeSpecName "kube-api-access-h4mns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:22.508593 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.508552    2565 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3800552f-1424-421a-aa4a-4a2c5f60cc98-cert\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:49:22.508593 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.508586    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4mns\" (UniqueName: \"kubernetes.io/projected/3800552f-1424-421a-aa4a-4a2c5f60cc98-kube-api-access-h4mns\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:49:22.686822 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:22.686795    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-557b59fdc4-f6qxg"]
Sep 16 18:49:22.688748 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:49:22.688722    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1126c57_2b37_4788_bb98_c043dde60b85.slice/crio-ae2e7301fb53030bd72b95df95f0fdfc3748d397b368bc7395dbb85a62f8ee97 WatchSource:0}: Error finding container ae2e7301fb53030bd72b95df95f0fdfc3748d397b368bc7395dbb85a62f8ee97: Status 404 returned error can't find the container with id ae2e7301fb53030bd72b95df95f0fdfc3748d397b368bc7395dbb85a62f8ee97
Sep 16 18:49:23.126509 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:23.126467    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-557b59fdc4-f6qxg" event={"ID":"c1126c57-2b37-4788-bb98-c043dde60b85","Type":"ContainerStarted","Data":"ae2e7301fb53030bd72b95df95f0fdfc3748d397b368bc7395dbb85a62f8ee97"}
Sep 16 18:49:23.127866 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:23.127834    2565 generic.go:356] "Generic (PLEG): container finished" podID="3800552f-1424-421a-aa4a-4a2c5f60cc98" containerID="879b9a76dcaab9485107606b2d553426365d580eee022ffe748d104090f5ae5a" exitCode=0
Sep 16 18:49:23.127987 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:23.127876    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-557b59fdc4-l8zmh"
Sep 16 18:49:23.127987 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:23.127882    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-557b59fdc4-l8zmh" event={"ID":"3800552f-1424-421a-aa4a-4a2c5f60cc98","Type":"ContainerDied","Data":"879b9a76dcaab9485107606b2d553426365d580eee022ffe748d104090f5ae5a"}
Sep 16 18:49:23.127987 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:23.127919    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-557b59fdc4-l8zmh" event={"ID":"3800552f-1424-421a-aa4a-4a2c5f60cc98","Type":"ContainerDied","Data":"d4e44fb9a57e84af54e7229347bebbdc3b304f9892fa6812a81e47778deb666e"}
Sep 16 18:49:23.127987 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:23.127941    2565 scope.go:117] "RemoveContainer" containerID="879b9a76dcaab9485107606b2d553426365d580eee022ffe748d104090f5ae5a"
Sep 16 18:49:23.150821 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:23.150795    2565 scope.go:117] "RemoveContainer" containerID="879b9a76dcaab9485107606b2d553426365d580eee022ffe748d104090f5ae5a"
Sep 16 18:49:23.152460 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:49:23.152420    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"879b9a76dcaab9485107606b2d553426365d580eee022ffe748d104090f5ae5a\": container with ID starting with 879b9a76dcaab9485107606b2d553426365d580eee022ffe748d104090f5ae5a not found: ID does not exist" containerID="879b9a76dcaab9485107606b2d553426365d580eee022ffe748d104090f5ae5a"
Sep 16 18:49:23.152577 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:23.152472    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"879b9a76dcaab9485107606b2d553426365d580eee022ffe748d104090f5ae5a"} err="failed to get container status \"879b9a76dcaab9485107606b2d553426365d580eee022ffe748d104090f5ae5a\": rpc error: code = NotFound desc = could not find container \"879b9a76dcaab9485107606b2d553426365d580eee022ffe748d104090f5ae5a\": container with ID starting with 879b9a76dcaab9485107606b2d553426365d580eee022ffe748d104090f5ae5a not found: ID does not exist"
Sep 16 18:49:23.165008 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:23.164968    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-557b59fdc4-l8zmh"]
Sep 16 18:49:23.172701 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:23.172678    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-557b59fdc4-l8zmh"]
Sep 16 18:49:24.136299 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:24.136257    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-557b59fdc4-f6qxg" event={"ID":"c1126c57-2b37-4788-bb98-c043dde60b85","Type":"ContainerStarted","Data":"52be5efed57dfaad02ce24b42344e9870161c547bbd41832431ff48c6a678957"}
Sep 16 18:49:24.136769 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:24.136388    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-557b59fdc4-f6qxg"
Sep 16 18:49:24.160958 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:24.160897    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-557b59fdc4-f6qxg" podStartSLOduration=1.747288486 podStartE2EDuration="2.160880888s" totalImagesPullingTime="413.592402ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:24.158519569 +0000 UTC m=+602.518037564" watchObservedRunningTime="2026-09-16 18:49:24.160880888 +0000 UTC m=+602.520398883"
Sep 16 18:49:24.208265 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:24.208229    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3800552f-1424-421a-aa4a-4a2c5f60cc98" path="/var/lib/kubelet/pods/3800552f-1424-421a-aa4a-4a2c5f60cc98/volumes"
Sep 16 18:49:55.147942 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:55.147854    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-557b59fdc4-f6qxg"
Sep 16 18:49:56.435580 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.435546    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-b56cx"]
Sep 16 18:49:56.435957 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.435944    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3800552f-1424-421a-aa4a-4a2c5f60cc98" containerName="manager"
Sep 16 18:49:56.435957 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.435956    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3800552f-1424-421a-aa4a-4a2c5f60cc98" containerName="manager"
Sep 16 18:49:56.436030 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.436015    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3800552f-1424-421a-aa4a-4a2c5f60cc98" containerName="manager"
Sep 16 18:49:56.442435 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.440792    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-b56cx"
Sep 16 18:49:56.443629 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.443595    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 16 18:49:56.443776 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.443699    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-5mfbm\""
Sep 16 18:49:56.456663 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.456629    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-k88vm"]
Sep 16 18:49:56.460172 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.460146    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-b56cx"]
Sep 16 18:49:56.460288 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.460251    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-k88vm"
Sep 16 18:49:56.463453 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.463428    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 16 18:49:56.463614 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.463593    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 16 18:49:56.463954 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.463935    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-h9mh7\""
Sep 16 18:49:56.473467 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.473432    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-k88vm"]
Sep 16 18:49:56.491104 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.491070    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ed332e-cd97-41ae-b43f-59c4b114d6b5-metrics-certs\") pod \"odh-model-controller-7b966d7984-k88vm\" (UID: \"e3ed332e-cd97-41ae-b43f-59c4b114d6b5\") " pod="kserve/odh-model-controller-7b966d7984-k88vm"
Sep 16 18:49:56.491330 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.491129    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tw6b\" (UniqueName: \"kubernetes.io/projected/e246356d-16a5-4654-b0f8-0c1f36c085c9-kube-api-access-4tw6b\") pod \"model-serving-api-7b954ffc95-b56cx\" (UID: \"e246356d-16a5-4654-b0f8-0c1f36c085c9\") " pod="kserve/model-serving-api-7b954ffc95-b56cx"
Sep 16 18:49:56.491330 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.491171    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7n8p\" (UniqueName: \"kubernetes.io/projected/e3ed332e-cd97-41ae-b43f-59c4b114d6b5-kube-api-access-p7n8p\") pod \"odh-model-controller-7b966d7984-k88vm\" (UID: \"e3ed332e-cd97-41ae-b43f-59c4b114d6b5\") " pod="kserve/odh-model-controller-7b966d7984-k88vm"
Sep 16 18:49:56.491330 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.491188    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e246356d-16a5-4654-b0f8-0c1f36c085c9-tls-certs\") pod \"model-serving-api-7b954ffc95-b56cx\" (UID: \"e246356d-16a5-4654-b0f8-0c1f36c085c9\") " pod="kserve/model-serving-api-7b954ffc95-b56cx"
Sep 16 18:49:56.491330 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.491275    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3ed332e-cd97-41ae-b43f-59c4b114d6b5-cert\") pod \"odh-model-controller-7b966d7984-k88vm\" (UID: \"e3ed332e-cd97-41ae-b43f-59c4b114d6b5\") " pod="kserve/odh-model-controller-7b966d7984-k88vm"
Sep 16 18:49:56.592338 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.592298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ed332e-cd97-41ae-b43f-59c4b114d6b5-metrics-certs\") pod \"odh-model-controller-7b966d7984-k88vm\" (UID: \"e3ed332e-cd97-41ae-b43f-59c4b114d6b5\") " pod="kserve/odh-model-controller-7b966d7984-k88vm"
Sep 16 18:49:56.592554 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.592370    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tw6b\" (UniqueName: \"kubernetes.io/projected/e246356d-16a5-4654-b0f8-0c1f36c085c9-kube-api-access-4tw6b\") pod \"model-serving-api-7b954ffc95-b56cx\" (UID: \"e246356d-16a5-4654-b0f8-0c1f36c085c9\") " pod="kserve/model-serving-api-7b954ffc95-b56cx"
Sep 16 18:49:56.592554 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.592409    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7n8p\" (UniqueName: \"kubernetes.io/projected/e3ed332e-cd97-41ae-b43f-59c4b114d6b5-kube-api-access-p7n8p\") pod \"odh-model-controller-7b966d7984-k88vm\" (UID: \"e3ed332e-cd97-41ae-b43f-59c4b114d6b5\") " pod="kserve/odh-model-controller-7b966d7984-k88vm"
Sep 16 18:49:56.592554 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.592426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e246356d-16a5-4654-b0f8-0c1f36c085c9-tls-certs\") pod \"model-serving-api-7b954ffc95-b56cx\" (UID: \"e246356d-16a5-4654-b0f8-0c1f36c085c9\") " pod="kserve/model-serving-api-7b954ffc95-b56cx"
Sep 16 18:49:56.592554 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.592466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3ed332e-cd97-41ae-b43f-59c4b114d6b5-cert\") pod \"odh-model-controller-7b966d7984-k88vm\" (UID: \"e3ed332e-cd97-41ae-b43f-59c4b114d6b5\") " pod="kserve/odh-model-controller-7b966d7984-k88vm"
Sep 16 18:49:56.592554 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:49:56.592541    2565 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 16 18:49:56.593024 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:49:56.592633    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e246356d-16a5-4654-b0f8-0c1f36c085c9-tls-certs podName:e246356d-16a5-4654-b0f8-0c1f36c085c9 nodeName:}" failed. No retries permitted until 2026-09-16 18:49:57.092611114 +0000 UTC m=+635.452129104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e246356d-16a5-4654-b0f8-0c1f36c085c9-tls-certs") pod "model-serving-api-7b954ffc95-b56cx" (UID: "e246356d-16a5-4654-b0f8-0c1f36c085c9") : secret "model-serving-api-tls" not found
Sep 16 18:49:56.593024 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:49:56.592549    2565 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 16 18:49:56.593024 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:49:56.592724    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3ed332e-cd97-41ae-b43f-59c4b114d6b5-cert podName:e3ed332e-cd97-41ae-b43f-59c4b114d6b5 nodeName:}" failed. No retries permitted until 2026-09-16 18:49:57.092710816 +0000 UTC m=+635.452228789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e3ed332e-cd97-41ae-b43f-59c4b114d6b5-cert") pod "odh-model-controller-7b966d7984-k88vm" (UID: "e3ed332e-cd97-41ae-b43f-59c4b114d6b5") : secret "odh-model-controller-webhook-cert" not found
Sep 16 18:49:56.595109 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.595084    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3ed332e-cd97-41ae-b43f-59c4b114d6b5-metrics-certs\") pod \"odh-model-controller-7b966d7984-k88vm\" (UID: \"e3ed332e-cd97-41ae-b43f-59c4b114d6b5\") " pod="kserve/odh-model-controller-7b966d7984-k88vm"
Sep 16 18:49:56.603937 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.603910    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tw6b\" (UniqueName: \"kubernetes.io/projected/e246356d-16a5-4654-b0f8-0c1f36c085c9-kube-api-access-4tw6b\") pod \"model-serving-api-7b954ffc95-b56cx\" (UID: \"e246356d-16a5-4654-b0f8-0c1f36c085c9\") " pod="kserve/model-serving-api-7b954ffc95-b56cx"
Sep 16 18:49:56.604067 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:56.603991    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7n8p\" (UniqueName: \"kubernetes.io/projected/e3ed332e-cd97-41ae-b43f-59c4b114d6b5-kube-api-access-p7n8p\") pod \"odh-model-controller-7b966d7984-k88vm\" (UID: \"e3ed332e-cd97-41ae-b43f-59c4b114d6b5\") " pod="kserve/odh-model-controller-7b966d7984-k88vm"
Sep 16 18:49:57.096625 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:57.096581    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3ed332e-cd97-41ae-b43f-59c4b114d6b5-cert\") pod \"odh-model-controller-7b966d7984-k88vm\" (UID: \"e3ed332e-cd97-41ae-b43f-59c4b114d6b5\") " pod="kserve/odh-model-controller-7b966d7984-k88vm"
Sep 16 18:49:57.096832 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:57.096680    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e246356d-16a5-4654-b0f8-0c1f36c085c9-tls-certs\") pod \"model-serving-api-7b954ffc95-b56cx\" (UID: \"e246356d-16a5-4654-b0f8-0c1f36c085c9\") " pod="kserve/model-serving-api-7b954ffc95-b56cx"
Sep 16 18:49:57.099158 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:57.099121    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e246356d-16a5-4654-b0f8-0c1f36c085c9-tls-certs\") pod \"model-serving-api-7b954ffc95-b56cx\" (UID: \"e246356d-16a5-4654-b0f8-0c1f36c085c9\") " pod="kserve/model-serving-api-7b954ffc95-b56cx"
Sep 16 18:49:57.099158 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:57.099121    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3ed332e-cd97-41ae-b43f-59c4b114d6b5-cert\") pod \"odh-model-controller-7b966d7984-k88vm\" (UID: \"e3ed332e-cd97-41ae-b43f-59c4b114d6b5\") " pod="kserve/odh-model-controller-7b966d7984-k88vm"
Sep 16 18:49:57.353997 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:57.353900    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-b56cx"
Sep 16 18:49:57.371742 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:57.371714    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-k88vm"
Sep 16 18:49:57.635758 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:57.635729    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-b56cx"]
Sep 16 18:49:57.637798 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:49:57.637756    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode246356d_16a5_4654_b0f8_0c1f36c085c9.slice/crio-6efba18d87cbf7f05a2e08d05ba1b6ddf2fe8e674458396e2b4b0fa9b146ca1e WatchSource:0}: Error finding container 6efba18d87cbf7f05a2e08d05ba1b6ddf2fe8e674458396e2b4b0fa9b146ca1e: Status 404 returned error can't find the container with id 6efba18d87cbf7f05a2e08d05ba1b6ddf2fe8e674458396e2b4b0fa9b146ca1e
Sep 16 18:49:57.639978 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:57.639958    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:49:57.657403 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:57.657377    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-k88vm"]
Sep 16 18:49:57.658924 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:49:57.658881    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ed332e_cd97_41ae_b43f_59c4b114d6b5.slice/crio-3077295b83780a9bcfabb98315aff06ef0f384491f4a6e08bbad37bda04b0aea WatchSource:0}: Error finding container 3077295b83780a9bcfabb98315aff06ef0f384491f4a6e08bbad37bda04b0aea: Status 404 returned error can't find the container with id 3077295b83780a9bcfabb98315aff06ef0f384491f4a6e08bbad37bda04b0aea
Sep 16 18:49:58.323245 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:58.323145    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-k88vm" event={"ID":"e3ed332e-cd97-41ae-b43f-59c4b114d6b5","Type":"ContainerStarted","Data":"3077295b83780a9bcfabb98315aff06ef0f384491f4a6e08bbad37bda04b0aea"}
Sep 16 18:49:58.326922 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:49:58.326852    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-b56cx" event={"ID":"e246356d-16a5-4654-b0f8-0c1f36c085c9","Type":"ContainerStarted","Data":"6efba18d87cbf7f05a2e08d05ba1b6ddf2fe8e674458396e2b4b0fa9b146ca1e"}
Sep 16 18:50:02.356620 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:50:02.356579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-b56cx" event={"ID":"e246356d-16a5-4654-b0f8-0c1f36c085c9","Type":"ContainerStarted","Data":"6dbb77ae9687b49de2c145cdbebcd9760528974ae98f0c22137f865fe421c341"}
Sep 16 18:50:02.357296 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:50:02.356668    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-b56cx"
Sep 16 18:50:02.358392 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:50:02.358365    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-k88vm" event={"ID":"e3ed332e-cd97-41ae-b43f-59c4b114d6b5","Type":"ContainerStarted","Data":"0a669fa2c2efd158283c0cc2c3ee8fe075f7dd05f8da90895a8befbf576eab88"}
Sep 16 18:50:02.358505 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:50:02.358494    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-k88vm"
Sep 16 18:50:02.374295 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:50:02.374240    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-b56cx" podStartSLOduration=2.726201203 podStartE2EDuration="6.374221258s" totalImagesPullingTime="3.648020055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:02.372563354 +0000 UTC m=+640.732081349" watchObservedRunningTime="2026-09-16 18:50:02.374221258 +0000 UTC m=+640.733739251"
Sep 16 18:50:02.394111 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:50:02.394051    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-k88vm" podStartSLOduration=2.715759339 podStartE2EDuration="6.394035117s" totalImagesPullingTime="3.678275778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:02.392374954 +0000 UTC m=+640.751892949" watchObservedRunningTime="2026-09-16 18:50:02.394035117 +0000 UTC m=+640.753553106"
Sep 16 18:50:13.366885 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:50:13.366853    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-k88vm"
Sep 16 18:50:13.368604 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:50:13.368579    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-b56cx"
Sep 16 18:52:18.493666 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:18.493623    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx"]
Sep 16 18:52:18.497376 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:18.497334    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx"
Sep 16 18:52:18.500922 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:18.500896    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 16 18:52:18.501984 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:18.501954    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-2grt6\""
Sep 16 18:52:18.502123 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:18.501997    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 16 18:52:18.502123 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:18.502012    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 16 18:52:18.502123 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:18.502082    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-wr9d6\""
Sep 16 18:52:18.511802 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:18.511774    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx"]
Sep 16 18:52:18.571579 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:18.571542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e002ea15-b9b4-446e-9982-8c2c289170a0-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx\" (UID: \"e002ea15-b9b4-446e-9982-8c2c289170a0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx"
Sep 16 18:52:18.571761 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:18.571637    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2t56\" (UniqueName: \"kubernetes.io/projected/e002ea15-b9b4-446e-9982-8c2c289170a0-kube-api-access-r2t56\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx\" (UID: \"e002ea15-b9b4-446e-9982-8c2c289170a0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx"
Sep 16 18:52:18.672215 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:18.672179    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2t56\" (UniqueName: \"kubernetes.io/projected/e002ea15-b9b4-446e-9982-8c2c289170a0-kube-api-access-r2t56\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx\" (UID: \"e002ea15-b9b4-446e-9982-8c2c289170a0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx"
Sep 16 18:52:18.672420 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:18.672269    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e002ea15-b9b4-446e-9982-8c2c289170a0-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx\" (UID: \"e002ea15-b9b4-446e-9982-8c2c289170a0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx"
Sep 16 18:52:18.675073 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:18.675043    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e002ea15-b9b4-446e-9982-8c2c289170a0-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx\" (UID: \"e002ea15-b9b4-446e-9982-8c2c289170a0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx"
Sep 16 18:52:18.691206 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:18.691170    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2t56\" (UniqueName: \"kubernetes.io/projected/e002ea15-b9b4-446e-9982-8c2c289170a0-kube-api-access-r2t56\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx\" (UID: \"e002ea15-b9b4-446e-9982-8c2c289170a0\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx"
Sep 16 18:52:18.808775 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:18.808660    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx"
Sep 16 18:52:19.091798 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:19.091760    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx"]
Sep 16 18:52:19.093752 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:52:19.093718    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode002ea15_b9b4_446e_9982_8c2c289170a0.slice/crio-ae76ab57a061f69dff7a2df69844da0709713d1f04c70a8fa8c7787ba17670b0 WatchSource:0}: Error finding container ae76ab57a061f69dff7a2df69844da0709713d1f04c70a8fa8c7787ba17670b0: Status 404 returned error can't find the container with id ae76ab57a061f69dff7a2df69844da0709713d1f04c70a8fa8c7787ba17670b0
Sep 16 18:52:19.153420 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:19.153380    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx" event={"ID":"e002ea15-b9b4-446e-9982-8c2c289170a0","Type":"ContainerStarted","Data":"ae76ab57a061f69dff7a2df69844da0709713d1f04c70a8fa8c7787ba17670b0"}
Sep 16 18:52:22.174407 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:22.174369    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx" event={"ID":"e002ea15-b9b4-446e-9982-8c2c289170a0","Type":"ContainerStarted","Data":"60010b9201aa4c533dc1de70a4d3a429b4261e3b2795f9b4db07c71265496b0d"}
Sep 16 18:52:22.174795 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:22.174474    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx"
Sep 16 18:52:22.194332 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:22.194269    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx" podStartSLOduration=1.768543984 podStartE2EDuration="4.194251516s" totalImagesPullingTime="2.425707532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:22.192943898 +0000 UTC m=+780.552461894" watchObservedRunningTime="2026-09-16 18:52:22.194251516 +0000 UTC m=+780.553769606"
Sep 16 18:52:53.182195 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:53.182119    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx"
Sep 16 18:52:59.984413 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:52:59.984296    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx"]
Sep 16 18:53:03.182183 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:03.182138    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx" podUID="e002ea15-b9b4-446e-9982-8c2c289170a0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:53:13.182708 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:13.182666    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx" podUID="e002ea15-b9b4-446e-9982-8c2c289170a0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:53:15.005937 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:15.005872    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx" podUID="e002ea15-b9b4-446e-9982-8c2c289170a0" containerName="main" containerID="cri-o://60010b9201aa4c533dc1de70a4d3a429b4261e3b2795f9b4db07c71265496b0d" gracePeriod=45
Sep 16 18:53:23.182149 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:23.182103    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx" podUID="e002ea15-b9b4-446e-9982-8c2c289170a0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:53:23.182687 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:23.182207    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx"
Sep 16 18:53:33.182837 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:33.182791    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx" podUID="e002ea15-b9b4-446e-9982-8c2c289170a0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:53:43.182165 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:43.182119    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx" podUID="e002ea15-b9b4-446e-9982-8c2c289170a0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:53:45.267368 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:45.267267    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx"
Sep 16 18:53:45.343486 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:45.343443    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e002ea15-b9b4-446e-9982-8c2c289170a0-tls-certs\") pod \"e002ea15-b9b4-446e-9982-8c2c289170a0\" (UID: \"e002ea15-b9b4-446e-9982-8c2c289170a0\") "
Sep 16 18:53:45.343695 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:45.343582    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2t56\" (UniqueName: \"kubernetes.io/projected/e002ea15-b9b4-446e-9982-8c2c289170a0-kube-api-access-r2t56\") pod \"e002ea15-b9b4-446e-9982-8c2c289170a0\" (UID: \"e002ea15-b9b4-446e-9982-8c2c289170a0\") "
Sep 16 18:53:45.345957 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:45.345915    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e002ea15-b9b4-446e-9982-8c2c289170a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e002ea15-b9b4-446e-9982-8c2c289170a0" (UID: "e002ea15-b9b4-446e-9982-8c2c289170a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:53:45.346077 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:45.345922    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e002ea15-b9b4-446e-9982-8c2c289170a0-kube-api-access-r2t56" (OuterVolumeSpecName: "kube-api-access-r2t56") pod "e002ea15-b9b4-446e-9982-8c2c289170a0" (UID: "e002ea15-b9b4-446e-9982-8c2c289170a0"). InnerVolumeSpecName "kube-api-access-r2t56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:53:45.445415 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:45.445325    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2t56\" (UniqueName: \"kubernetes.io/projected/e002ea15-b9b4-446e-9982-8c2c289170a0-kube-api-access-r2t56\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:53:45.445415 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:45.445410    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e002ea15-b9b4-446e-9982-8c2c289170a0-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:53:45.653900 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:45.653808    2565 generic.go:356] "Generic (PLEG): container finished" podID="e002ea15-b9b4-446e-9982-8c2c289170a0" containerID="60010b9201aa4c533dc1de70a4d3a429b4261e3b2795f9b4db07c71265496b0d" exitCode=0
Sep 16 18:53:45.653900 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:45.653854    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx"
Sep 16 18:53:45.653900 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:45.653892    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx" event={"ID":"e002ea15-b9b4-446e-9982-8c2c289170a0","Type":"ContainerDied","Data":"60010b9201aa4c533dc1de70a4d3a429b4261e3b2795f9b4db07c71265496b0d"}
Sep 16 18:53:45.654127 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:45.653944    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx" event={"ID":"e002ea15-b9b4-446e-9982-8c2c289170a0","Type":"ContainerDied","Data":"ae76ab57a061f69dff7a2df69844da0709713d1f04c70a8fa8c7787ba17670b0"}
Sep 16 18:53:45.654127 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:45.653968    2565 scope.go:117] "RemoveContainer" containerID="60010b9201aa4c533dc1de70a4d3a429b4261e3b2795f9b4db07c71265496b0d"
Sep 16 18:53:45.663789 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:45.663771    2565 scope.go:117] "RemoveContainer" containerID="60010b9201aa4c533dc1de70a4d3a429b4261e3b2795f9b4db07c71265496b0d"
Sep 16 18:53:45.664095 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:53:45.664078    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60010b9201aa4c533dc1de70a4d3a429b4261e3b2795f9b4db07c71265496b0d\": container with ID starting with 60010b9201aa4c533dc1de70a4d3a429b4261e3b2795f9b4db07c71265496b0d not found: ID does not exist" containerID="60010b9201aa4c533dc1de70a4d3a429b4261e3b2795f9b4db07c71265496b0d"
Sep 16 18:53:45.664157 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:45.664102    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60010b9201aa4c533dc1de70a4d3a429b4261e3b2795f9b4db07c71265496b0d"} err="failed to get container status \"60010b9201aa4c533dc1de70a4d3a429b4261e3b2795f9b4db07c71265496b0d\": rpc error: code = NotFound desc = could not find container \"60010b9201aa4c533dc1de70a4d3a429b4261e3b2795f9b4db07c71265496b0d\": container with ID starting with 60010b9201aa4c533dc1de70a4d3a429b4261e3b2795f9b4db07c71265496b0d not found: ID does not exist"
Sep 16 18:53:45.682067 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:45.682027    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx"]
Sep 16 18:53:45.685964 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:45.685932    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-596lx"]
Sep 16 18:53:46.207125 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:46.207090    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e002ea15-b9b4-446e-9982-8c2c289170a0" path="/var/lib/kubelet/pods/e002ea15-b9b4-446e-9982-8c2c289170a0/volumes"
Sep 16 18:53:56.373084 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:56.373044    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq"]
Sep 16 18:53:56.373795 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:56.373769    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e002ea15-b9b4-446e-9982-8c2c289170a0" containerName="main"
Sep 16 18:53:56.373881 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:56.373798    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e002ea15-b9b4-446e-9982-8c2c289170a0" containerName="main"
Sep 16 18:53:56.373938 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:56.373914    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e002ea15-b9b4-446e-9982-8c2c289170a0" containerName="main"
Sep 16 18:53:56.377154 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:56.377131    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq"
Sep 16 18:53:56.380232 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:56.380205    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 16 18:53:56.380473 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:56.380459    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 16 18:53:56.381155 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:56.381138    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-jm26z\""
Sep 16 18:53:56.381278 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:56.381258    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 16 18:53:56.381375 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:56.381297    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-2swtq\""
Sep 16 18:53:56.389094 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:56.389064    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq"]
Sep 16 18:53:56.544794 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:56.544760    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06337c66-ec59-4cd6-8e07-24e3af1860ab-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq\" (UID: \"06337c66-ec59-4cd6-8e07-24e3af1860ab\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq"
Sep 16 18:53:56.544993 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:56.544891    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d52fz\" (UniqueName: \"kubernetes.io/projected/06337c66-ec59-4cd6-8e07-24e3af1860ab-kube-api-access-d52fz\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq\" (UID: \"06337c66-ec59-4cd6-8e07-24e3af1860ab\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq"
Sep 16 18:53:56.646074 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:56.645976    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d52fz\" (UniqueName: \"kubernetes.io/projected/06337c66-ec59-4cd6-8e07-24e3af1860ab-kube-api-access-d52fz\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq\" (UID: \"06337c66-ec59-4cd6-8e07-24e3af1860ab\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq"
Sep 16 18:53:56.646074 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:56.646038    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06337c66-ec59-4cd6-8e07-24e3af1860ab-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq\" (UID: \"06337c66-ec59-4cd6-8e07-24e3af1860ab\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq"
Sep 16 18:53:56.648662 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:56.648628    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06337c66-ec59-4cd6-8e07-24e3af1860ab-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq\" (UID: \"06337c66-ec59-4cd6-8e07-24e3af1860ab\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq"
Sep 16 18:53:56.656105 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:56.656077    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d52fz\" (UniqueName: \"kubernetes.io/projected/06337c66-ec59-4cd6-8e07-24e3af1860ab-kube-api-access-d52fz\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq\" (UID: \"06337c66-ec59-4cd6-8e07-24e3af1860ab\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq"
Sep 16 18:53:56.689892 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:56.689850    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq"
Sep 16 18:53:56.967447 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:56.967420    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq"]
Sep 16 18:53:56.969969 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:53:56.969937    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06337c66_ec59_4cd6_8e07_24e3af1860ab.slice/crio-8a51aa4bc087b60ede82f9e88655096d3e9323b31aeb5a3fff7cdc0416c5d86b WatchSource:0}: Error finding container 8a51aa4bc087b60ede82f9e88655096d3e9323b31aeb5a3fff7cdc0416c5d86b: Status 404 returned error can't find the container with id 8a51aa4bc087b60ede82f9e88655096d3e9323b31aeb5a3fff7cdc0416c5d86b
Sep 16 18:53:57.734788 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:57.734749    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq" event={"ID":"06337c66-ec59-4cd6-8e07-24e3af1860ab","Type":"ContainerStarted","Data":"48b03085d5e47879b2d5340d8e8910d7a6fdc152e4f871e378008a3af44e7c86"}
Sep 16 18:53:57.735210 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:57.734813    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq" event={"ID":"06337c66-ec59-4cd6-8e07-24e3af1860ab","Type":"ContainerStarted","Data":"8a51aa4bc087b60ede82f9e88655096d3e9323b31aeb5a3fff7cdc0416c5d86b"}
Sep 16 18:53:57.735210 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:57.734853    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq"
Sep 16 18:53:57.760564 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:53:57.760511    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq" podStartSLOduration=1.7604946510000001 podStartE2EDuration="1.760494651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:57.757790588 +0000 UTC m=+876.117308584" watchObservedRunningTime="2026-09-16 18:53:57.760494651 +0000 UTC m=+876.120012645"
Sep 16 18:54:28.742445 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:54:28.742412    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq"
Sep 16 18:54:35.186747 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:54:35.186705    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq"]
Sep 16 18:54:38.742294 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:54:38.742249    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq" podUID="06337c66-ec59-4cd6-8e07-24e3af1860ab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:54:48.742706 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:54:48.742659    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq" podUID="06337c66-ec59-4cd6-8e07-24e3af1860ab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:54:50.208637 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:54:50.208582    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq" podUID="06337c66-ec59-4cd6-8e07-24e3af1860ab" containerName="main" containerID="cri-o://48b03085d5e47879b2d5340d8e8910d7a6fdc152e4f871e378008a3af44e7c86" gracePeriod=45
Sep 16 18:54:58.742621 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:54:58.742574    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq" podUID="06337c66-ec59-4cd6-8e07-24e3af1860ab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:54:58.744581 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:54:58.742693    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq"
Sep 16 18:55:05.333622 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:05.333585    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq"]
Sep 16 18:55:05.337565 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:05.337537    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq"
Sep 16 18:55:05.343090 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:05.343047    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-cx5b4\""
Sep 16 18:55:05.343090 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:05.343082    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 18:55:05.343383 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:05.343082    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 18:55:05.343383 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:05.343082    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-2jgc6\""
Sep 16 18:55:05.343383 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:05.343192    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 18:55:05.351872 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:05.351838    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq"]
Sep 16 18:55:05.357965 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:05.357934    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq\" (UID: \"6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq"
Sep 16 18:55:05.357965 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:05.357969    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpbff\" (UniqueName: \"kubernetes.io/projected/6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6-kube-api-access-vpbff\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq\" (UID: \"6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq"
Sep 16 18:55:05.458793 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:05.458744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq\" (UID: \"6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq"
Sep 16 18:55:05.458793 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:05.458791    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpbff\" (UniqueName: \"kubernetes.io/projected/6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6-kube-api-access-vpbff\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq\" (UID: \"6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq"
Sep 16 18:55:05.461607 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:05.461576    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq\" (UID: \"6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq"
Sep 16 18:55:05.470429 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:05.470401    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpbff\" (UniqueName: \"kubernetes.io/projected/6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6-kube-api-access-vpbff\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq\" (UID: \"6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq"
Sep 16 18:55:05.648689 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:05.648584    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq"
Sep 16 18:55:05.949330 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:05.949298    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq"]
Sep 16 18:55:05.951268 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:55:05.951238    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1e5d9b_5642_4f5d_b459_d1965d7f8ca6.slice/crio-106d3ff4096c0771ae580ca0aa35873787105f848eb413a8c7b96e2fffbab709 WatchSource:0}: Error finding container 106d3ff4096c0771ae580ca0aa35873787105f848eb413a8c7b96e2fffbab709: Status 404 returned error can't find the container with id 106d3ff4096c0771ae580ca0aa35873787105f848eb413a8c7b96e2fffbab709
Sep 16 18:55:06.131133 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:06.131093    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq" event={"ID":"6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6","Type":"ContainerStarted","Data":"11c104794396eba438ddf23c01f368c22aa1fca9200c52654a4b823067ebf2a1"}
Sep 16 18:55:06.131325 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:06.131148    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq" event={"ID":"6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6","Type":"ContainerStarted","Data":"106d3ff4096c0771ae580ca0aa35873787105f848eb413a8c7b96e2fffbab709"}
Sep 16 18:55:06.131325 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:06.131204    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq"
Sep 16 18:55:06.154109 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:06.154045    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq" podStartSLOduration=1.154027973 podStartE2EDuration="1.154027973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:06.153497819 +0000 UTC m=+944.513015815" watchObservedRunningTime="2026-09-16 18:55:06.154027973 +0000 UTC m=+944.513546025"
Sep 16 18:55:08.742536 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:08.742498    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq" podUID="06337c66-ec59-4cd6-8e07-24e3af1860ab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:55:18.742002 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:18.741961    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq" podUID="06337c66-ec59-4cd6-8e07-24e3af1860ab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:55:20.467592 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:20.467514    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq"
Sep 16 18:55:20.604248 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:20.604214    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06337c66-ec59-4cd6-8e07-24e3af1860ab-tls-certs\") pod \"06337c66-ec59-4cd6-8e07-24e3af1860ab\" (UID: \"06337c66-ec59-4cd6-8e07-24e3af1860ab\") "
Sep 16 18:55:20.604459 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:20.604303    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d52fz\" (UniqueName: \"kubernetes.io/projected/06337c66-ec59-4cd6-8e07-24e3af1860ab-kube-api-access-d52fz\") pod \"06337c66-ec59-4cd6-8e07-24e3af1860ab\" (UID: \"06337c66-ec59-4cd6-8e07-24e3af1860ab\") "
Sep 16 18:55:20.606584 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:20.606550    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06337c66-ec59-4cd6-8e07-24e3af1860ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06337c66-ec59-4cd6-8e07-24e3af1860ab" (UID: "06337c66-ec59-4cd6-8e07-24e3af1860ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:20.606701 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:20.606657    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06337c66-ec59-4cd6-8e07-24e3af1860ab-kube-api-access-d52fz" (OuterVolumeSpecName: "kube-api-access-d52fz") pod "06337c66-ec59-4cd6-8e07-24e3af1860ab" (UID: "06337c66-ec59-4cd6-8e07-24e3af1860ab"). InnerVolumeSpecName "kube-api-access-d52fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:20.705805 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:20.705763    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d52fz\" (UniqueName: \"kubernetes.io/projected/06337c66-ec59-4cd6-8e07-24e3af1860ab-kube-api-access-d52fz\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:55:20.705805 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:20.705797    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06337c66-ec59-4cd6-8e07-24e3af1860ab-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:55:21.221802 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:21.221760    2565 generic.go:356] "Generic (PLEG): container finished" podID="06337c66-ec59-4cd6-8e07-24e3af1860ab" containerID="48b03085d5e47879b2d5340d8e8910d7a6fdc152e4f871e378008a3af44e7c86" exitCode=0
Sep 16 18:55:21.222011 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:21.221827    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq"
Sep 16 18:55:21.222011 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:21.221844    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq" event={"ID":"06337c66-ec59-4cd6-8e07-24e3af1860ab","Type":"ContainerDied","Data":"48b03085d5e47879b2d5340d8e8910d7a6fdc152e4f871e378008a3af44e7c86"}
Sep 16 18:55:21.222011 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:21.221926    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq" event={"ID":"06337c66-ec59-4cd6-8e07-24e3af1860ab","Type":"ContainerDied","Data":"8a51aa4bc087b60ede82f9e88655096d3e9323b31aeb5a3fff7cdc0416c5d86b"}
Sep 16 18:55:21.222011 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:21.221971    2565 scope.go:117] "RemoveContainer" containerID="48b03085d5e47879b2d5340d8e8910d7a6fdc152e4f871e378008a3af44e7c86"
Sep 16 18:55:21.231911 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:21.231886    2565 scope.go:117] "RemoveContainer" containerID="48b03085d5e47879b2d5340d8e8910d7a6fdc152e4f871e378008a3af44e7c86"
Sep 16 18:55:21.232256 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:55:21.232233    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48b03085d5e47879b2d5340d8e8910d7a6fdc152e4f871e378008a3af44e7c86\": container with ID starting with 48b03085d5e47879b2d5340d8e8910d7a6fdc152e4f871e378008a3af44e7c86 not found: ID does not exist" containerID="48b03085d5e47879b2d5340d8e8910d7a6fdc152e4f871e378008a3af44e7c86"
Sep 16 18:55:21.232334 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:21.232270    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b03085d5e47879b2d5340d8e8910d7a6fdc152e4f871e378008a3af44e7c86"} err="failed to get container status \"48b03085d5e47879b2d5340d8e8910d7a6fdc152e4f871e378008a3af44e7c86\": rpc error: code = NotFound desc = could not find container \"48b03085d5e47879b2d5340d8e8910d7a6fdc152e4f871e378008a3af44e7c86\": container with ID starting with 48b03085d5e47879b2d5340d8e8910d7a6fdc152e4f871e378008a3af44e7c86 not found: ID does not exist"
Sep 16 18:55:21.247409 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:21.247364    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq"]
Sep 16 18:55:21.249698 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:21.249676    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-4gsbq"]
Sep 16 18:55:22.207447 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:22.207416    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06337c66-ec59-4cd6-8e07-24e3af1860ab" path="/var/lib/kubelet/pods/06337c66-ec59-4cd6-8e07-24e3af1860ab/volumes"
Sep 16 18:55:31.346558 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.346521    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"]
Sep 16 18:55:31.346986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.346966    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06337c66-ec59-4cd6-8e07-24e3af1860ab" containerName="main"
Sep 16 18:55:31.346986 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.346982    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="06337c66-ec59-4cd6-8e07-24e3af1860ab" containerName="main"
Sep 16 18:55:31.347076 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.347066    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="06337c66-ec59-4cd6-8e07-24e3af1860ab" containerName="main"
Sep 16 18:55:31.351682 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.351660    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.354203 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.354177    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 16 18:55:31.355101 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.355069    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 16 18:55:31.355251 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.355122    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-p6sjs\""
Sep 16 18:55:31.355251 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.355155    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 16 18:55:31.360658 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.359232    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"]
Sep 16 18:55:31.503322 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.503268    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.503550 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.503424    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.503550 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.503465    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.503550 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.503484    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9dc6\" (UniqueName: \"kubernetes.io/projected/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-kube-api-access-s9dc6\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.503550 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.503513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.503550 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.503532    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.604209 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.604096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.604209 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.604186    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.604209 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.604211    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.604555 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.604234    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9dc6\" (UniqueName: \"kubernetes.io/projected/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-kube-api-access-s9dc6\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.604555 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.604271    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.604555 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.604301    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.604734 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.604709    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.604807 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.604721    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.604807 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.604779    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.606862 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.606821    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.607101 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.607079    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.613142 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.613108    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9dc6\" (UniqueName: \"kubernetes.io/projected/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-kube-api-access-s9dc6\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.670110 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.670066    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:31.953616 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.953586    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"]
Sep 16 18:55:31.955193 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:55:31.955152    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7d4d83_68e6_4de4_a9a9_4c54bd76773c.slice/crio-ecbf0293b49a41ad74bb9793bcc4932fa7fec6a0698d47bdfad42cb60c557684 WatchSource:0}: Error finding container ecbf0293b49a41ad74bb9793bcc4932fa7fec6a0698d47bdfad42cb60c557684: Status 404 returned error can't find the container with id ecbf0293b49a41ad74bb9793bcc4932fa7fec6a0698d47bdfad42cb60c557684
Sep 16 18:55:31.957176 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:31.957149    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:55:32.285240 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:32.285191    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98" event={"ID":"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c","Type":"ContainerStarted","Data":"ecbf0293b49a41ad74bb9793bcc4932fa7fec6a0698d47bdfad42cb60c557684"}
Sep 16 18:55:34.299520 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:34.299478    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98" event={"ID":"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c","Type":"ContainerStarted","Data":"6bb5e1742db0344babc230d5573bf75e0c86b4e72cd8917632168ba746476bb8"}
Sep 16 18:55:34.445050 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:34.444980    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98" podStartSLOduration=1.779590893 podStartE2EDuration="3.444961353s" totalImagesPullingTime="1.66537046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:34.441681874 +0000 UTC m=+972.801199868" watchObservedRunningTime="2026-09-16 18:55:34.444961353 +0000 UTC m=+972.804479347"
Sep 16 18:55:34.670692 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:34.670580    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:37.139417 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:37.139375    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq"
Sep 16 18:55:38.038257 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.038212    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"]
Sep 16 18:55:38.048169 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:55:38.048128    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:55:38.048169 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:55:38.048169 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 18:55:38.048169 ip-10-0-130-204 kubenswrapper[2565]:  > containerID="6bb5e1742db0344babc230d5573bf75e0c86b4e72cd8917632168ba746476bb8" cmd=["/bin/sleep","15"]
Sep 16 18:55:38.048519 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.048192    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98" message="" err=<
Sep 16 18:55:38.048519 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:55:38.048519 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 18:55:38.048519 ip-10-0-130-204 kubenswrapper[2565]:  >
Sep 16 18:55:38.048519 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:55:38.048206    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:55:38.048519 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:55:38.048519 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 18:55:38.048519 ip-10-0-130-204 kubenswrapper[2565]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98" podUID="7c7d4d83-68e6-4de4-a9a9-4c54bd76773c" containerName="main" containerID="cri-o://6bb5e1742db0344babc230d5573bf75e0c86b4e72cd8917632168ba746476bb8"
Sep 16 18:55:38.048519 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.048248    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98" podUID="7c7d4d83-68e6-4de4-a9a9-4c54bd76773c" containerName="main" containerID="cri-o://6bb5e1742db0344babc230d5573bf75e0c86b4e72cd8917632168ba746476bb8" gracePeriod=60
Sep 16 18:55:38.315378 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.315329    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:38.324389 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.324333    2565 generic.go:356] "Generic (PLEG): container finished" podID="7c7d4d83-68e6-4de4-a9a9-4c54bd76773c" containerID="6bb5e1742db0344babc230d5573bf75e0c86b4e72cd8917632168ba746476bb8" exitCode=0
Sep 16 18:55:38.324587 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.324398    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"
Sep 16 18:55:38.324587 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.324416    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98" event={"ID":"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c","Type":"ContainerDied","Data":"6bb5e1742db0344babc230d5573bf75e0c86b4e72cd8917632168ba746476bb8"}
Sep 16 18:55:38.324587 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.324488    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98" event={"ID":"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c","Type":"ContainerDied","Data":"ecbf0293b49a41ad74bb9793bcc4932fa7fec6a0698d47bdfad42cb60c557684"}
Sep 16 18:55:38.324587 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.324519    2565 scope.go:117] "RemoveContainer" containerID="6bb5e1742db0344babc230d5573bf75e0c86b4e72cd8917632168ba746476bb8"
Sep 16 18:55:38.334373 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.334325    2565 scope.go:117] "RemoveContainer" containerID="6bb5e1742db0344babc230d5573bf75e0c86b4e72cd8917632168ba746476bb8"
Sep 16 18:55:38.334746 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:55:38.334716    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb5e1742db0344babc230d5573bf75e0c86b4e72cd8917632168ba746476bb8\": container with ID starting with 6bb5e1742db0344babc230d5573bf75e0c86b4e72cd8917632168ba746476bb8 not found: ID does not exist" containerID="6bb5e1742db0344babc230d5573bf75e0c86b4e72cd8917632168ba746476bb8"
Sep 16 18:55:38.334850 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.334758    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb5e1742db0344babc230d5573bf75e0c86b4e72cd8917632168ba746476bb8"} err="failed to get container status \"6bb5e1742db0344babc230d5573bf75e0c86b4e72cd8917632168ba746476bb8\": rpc error: code = NotFound desc = could not find container \"6bb5e1742db0344babc230d5573bf75e0c86b4e72cd8917632168ba746476bb8\": container with ID starting with 6bb5e1742db0344babc230d5573bf75e0c86b4e72cd8917632168ba746476bb8 not found: ID does not exist"
Sep 16 18:55:38.471014 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.470968    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9dc6\" (UniqueName: \"kubernetes.io/projected/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-kube-api-access-s9dc6\") pod \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") "
Sep 16 18:55:38.471217 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.471060    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-tls-certs\") pod \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") "
Sep 16 18:55:38.471217 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.471107    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-model-cache\") pod \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") "
Sep 16 18:55:38.471217 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.471127    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-dshm\") pod \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") "
Sep 16 18:55:38.471217 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.471148    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-tmp-dir\") pod \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") "
Sep 16 18:55:38.471217 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.471171    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-home\") pod \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\" (UID: \"7c7d4d83-68e6-4de4-a9a9-4c54bd76773c\") "
Sep 16 18:55:38.471537 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.471473    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c7d4d83-68e6-4de4-a9a9-4c54bd76773c" (UID: "7c7d4d83-68e6-4de4-a9a9-4c54bd76773c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:38.471537 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.471486    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c7d4d83-68e6-4de4-a9a9-4c54bd76773c" (UID: "7c7d4d83-68e6-4de4-a9a9-4c54bd76773c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:38.471615 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.471531    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-home" (OuterVolumeSpecName: "home") pod "7c7d4d83-68e6-4de4-a9a9-4c54bd76773c" (UID: "7c7d4d83-68e6-4de4-a9a9-4c54bd76773c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:38.473530 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.473503    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-dshm" (OuterVolumeSpecName: "dshm") pod "7c7d4d83-68e6-4de4-a9a9-4c54bd76773c" (UID: "7c7d4d83-68e6-4de4-a9a9-4c54bd76773c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:38.473620 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.473577    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c7d4d83-68e6-4de4-a9a9-4c54bd76773c" (UID: "7c7d4d83-68e6-4de4-a9a9-4c54bd76773c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:38.473683 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.473662    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-kube-api-access-s9dc6" (OuterVolumeSpecName: "kube-api-access-s9dc6") pod "7c7d4d83-68e6-4de4-a9a9-4c54bd76773c" (UID: "7c7d4d83-68e6-4de4-a9a9-4c54bd76773c"). InnerVolumeSpecName "kube-api-access-s9dc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:38.572425 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.572298    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:55:38.572425 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.572334    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9dc6\" (UniqueName: \"kubernetes.io/projected/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-kube-api-access-s9dc6\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:55:38.572425 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.572373    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:55:38.572425 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.572383    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:55:38.572425 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.572392    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:55:38.572425 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.572399    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:55:38.649504 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.649461    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"]
Sep 16 18:55:38.654682 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:38.654649    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-9fm98"]
Sep 16 18:55:40.208531 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:40.208499    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c7d4d83-68e6-4de4-a9a9-4c54bd76773c" path="/var/lib/kubelet/pods/7c7d4d83-68e6-4de4-a9a9-4c54bd76773c/volumes"
Sep 16 18:55:49.226049 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.225999    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"]
Sep 16 18:55:49.226652 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.226629    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c7d4d83-68e6-4de4-a9a9-4c54bd76773c" containerName="main"
Sep 16 18:55:49.226652 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.226655    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c7d4d83-68e6-4de4-a9a9-4c54bd76773c" containerName="main"
Sep 16 18:55:49.226804 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.226759    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c7d4d83-68e6-4de4-a9a9-4c54bd76773c" containerName="main"
Sep 16 18:55:49.229970 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.229947    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.234183 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.234156    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 16 18:55:49.234624 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.234602    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 16 18:55:49.235063 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.235041    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-v2frm\""
Sep 16 18:55:49.235198 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.235106    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 16 18:55:49.241988 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.241959    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"]
Sep 16 18:55:49.266127 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.266090    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.266127 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.266140    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.266417 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.266239    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.266417 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.266279    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47ca6111-25d4-43ae-ac99-9c335d044869-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.266417 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.266371    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.266417 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.266397    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk6c7\" (UniqueName: \"kubernetes.io/projected/47ca6111-25d4-43ae-ac99-9c335d044869-kube-api-access-nk6c7\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.367754 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.367714    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.367754 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.367763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47ca6111-25d4-43ae-ac99-9c335d044869-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.367995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.367794    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.367995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.367823    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk6c7\" (UniqueName: \"kubernetes.io/projected/47ca6111-25d4-43ae-ac99-9c335d044869-kube-api-access-nk6c7\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.367995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.367846    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.367995 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.367874    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.368314 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.368288    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.368446 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.368335    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.368505 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.368470    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.370657 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.370614    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.370931 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.370909    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47ca6111-25d4-43ae-ac99-9c335d044869-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.376545 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.376523    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk6c7\" (UniqueName: \"kubernetes.io/projected/47ca6111-25d4-43ae-ac99-9c335d044869-kube-api-access-nk6c7\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.544696 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.544598    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:49.841613 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:49.841578    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"]
Sep 16 18:55:49.843624 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:55:49.843583    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ca6111_25d4_43ae_ac99_9c335d044869.slice/crio-083fae9a9e22d25df8123d7536c23e6ce923d4adf7fb4a45796b1a2ff01785ed WatchSource:0}: Error finding container 083fae9a9e22d25df8123d7536c23e6ce923d4adf7fb4a45796b1a2ff01785ed: Status 404 returned error can't find the container with id 083fae9a9e22d25df8123d7536c23e6ce923d4adf7fb4a45796b1a2ff01785ed
Sep 16 18:55:50.394152 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:50.394112    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5" event={"ID":"47ca6111-25d4-43ae-ac99-9c335d044869","Type":"ContainerStarted","Data":"ce2732dce34f0da70f406f89c6032e4f5a200dbaa52d2580c41aaac2a2ba3477"}
Sep 16 18:55:50.394570 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:50.394175    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5" event={"ID":"47ca6111-25d4-43ae-ac99-9c335d044869","Type":"ContainerStarted","Data":"083fae9a9e22d25df8123d7536c23e6ce923d4adf7fb4a45796b1a2ff01785ed"}
Sep 16 18:55:50.420040 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:50.419979    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5" podStartSLOduration=1.419961585 podStartE2EDuration="1.419961585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:50.418020945 +0000 UTC m=+988.777538953" watchObservedRunningTime="2026-09-16 18:55:50.419961585 +0000 UTC m=+988.779479580"
Sep 16 18:55:50.545394 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:50.545340    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:53.384711 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.384668    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"]
Sep 16 18:55:53.427123 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:55:53.427075    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:55:53.427123 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:55:53.427123 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 18:55:53.427123 ip-10-0-130-204 kubenswrapper[2565]:  > containerID="ce2732dce34f0da70f406f89c6032e4f5a200dbaa52d2580c41aaac2a2ba3477" cmd=["/bin/sleep","15"]
Sep 16 18:55:53.427484 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.427134    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5" message="" err=<
Sep 16 18:55:53.427484 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:55:53.427484 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 18:55:53.427484 ip-10-0-130-204 kubenswrapper[2565]:  >
Sep 16 18:55:53.427484 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:55:53.427147    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:55:53.427484 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:55:53.427484 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 18:55:53.427484 ip-10-0-130-204 kubenswrapper[2565]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5" podUID="47ca6111-25d4-43ae-ac99-9c335d044869" containerName="main" containerID="cri-o://ce2732dce34f0da70f406f89c6032e4f5a200dbaa52d2580c41aaac2a2ba3477"
Sep 16 18:55:53.427484 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.427195    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5" podUID="47ca6111-25d4-43ae-ac99-9c335d044869" containerName="main" containerID="cri-o://ce2732dce34f0da70f406f89c6032e4f5a200dbaa52d2580c41aaac2a2ba3477" gracePeriod=60
Sep 16 18:55:53.689996 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.689967    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:53.804829 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.804788    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-home\") pod \"47ca6111-25d4-43ae-ac99-9c335d044869\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") "
Sep 16 18:55:53.804829 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.804835    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-tmp-dir\") pod \"47ca6111-25d4-43ae-ac99-9c335d044869\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") "
Sep 16 18:55:53.805106 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.804867    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-dshm\") pod \"47ca6111-25d4-43ae-ac99-9c335d044869\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") "
Sep 16 18:55:53.805106 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.804972    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47ca6111-25d4-43ae-ac99-9c335d044869-tls-certs\") pod \"47ca6111-25d4-43ae-ac99-9c335d044869\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") "
Sep 16 18:55:53.805106 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.805032    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-model-cache\") pod \"47ca6111-25d4-43ae-ac99-9c335d044869\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") "
Sep 16 18:55:53.805106 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.805076    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nk6c7\" (UniqueName: \"kubernetes.io/projected/47ca6111-25d4-43ae-ac99-9c335d044869-kube-api-access-nk6c7\") pod \"47ca6111-25d4-43ae-ac99-9c335d044869\" (UID: \"47ca6111-25d4-43ae-ac99-9c335d044869\") "
Sep 16 18:55:53.805305 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.805104    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "47ca6111-25d4-43ae-ac99-9c335d044869" (UID: "47ca6111-25d4-43ae-ac99-9c335d044869"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:53.805305 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.805116    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-home" (OuterVolumeSpecName: "home") pod "47ca6111-25d4-43ae-ac99-9c335d044869" (UID: "47ca6111-25d4-43ae-ac99-9c335d044869"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:53.805444 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.805301    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-model-cache" (OuterVolumeSpecName: "model-cache") pod "47ca6111-25d4-43ae-ac99-9c335d044869" (UID: "47ca6111-25d4-43ae-ac99-9c335d044869"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:53.805444 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.805336    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:55:53.805444 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.805389    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:55:53.807243 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.807214    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ca6111-25d4-43ae-ac99-9c335d044869-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47ca6111-25d4-43ae-ac99-9c335d044869" (UID: "47ca6111-25d4-43ae-ac99-9c335d044869"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:53.807358 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.807220    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-dshm" (OuterVolumeSpecName: "dshm") pod "47ca6111-25d4-43ae-ac99-9c335d044869" (UID: "47ca6111-25d4-43ae-ac99-9c335d044869"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:53.807358 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.807321    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ca6111-25d4-43ae-ac99-9c335d044869-kube-api-access-nk6c7" (OuterVolumeSpecName: "kube-api-access-nk6c7") pod "47ca6111-25d4-43ae-ac99-9c335d044869" (UID: "47ca6111-25d4-43ae-ac99-9c335d044869"). InnerVolumeSpecName "kube-api-access-nk6c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:53.905905 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.905860    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47ca6111-25d4-43ae-ac99-9c335d044869-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:55:53.905905 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.905896    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:55:53.905905 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.905910    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk6c7\" (UniqueName: \"kubernetes.io/projected/47ca6111-25d4-43ae-ac99-9c335d044869-kube-api-access-nk6c7\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:55:53.906145 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:53.905925    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47ca6111-25d4-43ae-ac99-9c335d044869-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:55:54.425102 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:54.425066    2565 generic.go:356] "Generic (PLEG): container finished" podID="47ca6111-25d4-43ae-ac99-9c335d044869" containerID="ce2732dce34f0da70f406f89c6032e4f5a200dbaa52d2580c41aaac2a2ba3477" exitCode=0
Sep 16 18:55:54.425569 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:54.425115    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5" event={"ID":"47ca6111-25d4-43ae-ac99-9c335d044869","Type":"ContainerDied","Data":"ce2732dce34f0da70f406f89c6032e4f5a200dbaa52d2580c41aaac2a2ba3477"}
Sep 16 18:55:54.425569 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:54.425123    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"
Sep 16 18:55:54.425569 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:54.425159    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5" event={"ID":"47ca6111-25d4-43ae-ac99-9c335d044869","Type":"ContainerDied","Data":"083fae9a9e22d25df8123d7536c23e6ce923d4adf7fb4a45796b1a2ff01785ed"}
Sep 16 18:55:54.425569 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:54.425180    2565 scope.go:117] "RemoveContainer" containerID="ce2732dce34f0da70f406f89c6032e4f5a200dbaa52d2580c41aaac2a2ba3477"
Sep 16 18:55:54.435137 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:54.434945    2565 scope.go:117] "RemoveContainer" containerID="ce2732dce34f0da70f406f89c6032e4f5a200dbaa52d2580c41aaac2a2ba3477"
Sep 16 18:55:54.435312 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:55:54.435290    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2732dce34f0da70f406f89c6032e4f5a200dbaa52d2580c41aaac2a2ba3477\": container with ID starting with ce2732dce34f0da70f406f89c6032e4f5a200dbaa52d2580c41aaac2a2ba3477 not found: ID does not exist" containerID="ce2732dce34f0da70f406f89c6032e4f5a200dbaa52d2580c41aaac2a2ba3477"
Sep 16 18:55:54.435384 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:54.435337    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2732dce34f0da70f406f89c6032e4f5a200dbaa52d2580c41aaac2a2ba3477"} err="failed to get container status \"ce2732dce34f0da70f406f89c6032e4f5a200dbaa52d2580c41aaac2a2ba3477\": rpc error: code = NotFound desc = could not find container \"ce2732dce34f0da70f406f89c6032e4f5a200dbaa52d2580c41aaac2a2ba3477\": container with ID starting with ce2732dce34f0da70f406f89c6032e4f5a200dbaa52d2580c41aaac2a2ba3477 not found: ID does not exist"
Sep 16 18:55:54.456055 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:54.456020    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"]
Sep 16 18:55:54.458235 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:54.458208    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvm8m5"]
Sep 16 18:55:56.208077 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:55:56.208038    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47ca6111-25d4-43ae-ac99-9c335d044869" path="/var/lib/kubelet/pods/47ca6111-25d4-43ae-ac99-9c335d044869/volumes"
Sep 16 18:56:13.204043 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:13.204004    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6"]
Sep 16 18:56:13.204512 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:13.204439    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ca6111-25d4-43ae-ac99-9c335d044869" containerName="main"
Sep 16 18:56:13.204512 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:13.204455    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ca6111-25d4-43ae-ac99-9c335d044869" containerName="main"
Sep 16 18:56:13.204606 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:13.204548    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="47ca6111-25d4-43ae-ac99-9c335d044869" containerName="main"
Sep 16 18:56:13.212747 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:13.212716    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6"
Sep 16 18:56:13.216109 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:13.216079    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 16 18:56:13.216319 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:13.216276    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-t6526\""
Sep 16 18:56:13.216440 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:13.216337    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 16 18:56:13.216440 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:13.216375    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 16 18:56:13.216552 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:13.216527    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-z5rh6\""
Sep 16 18:56:13.217397 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:13.217367    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6"]
Sep 16 18:56:13.390964 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:13.390923    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xslrb\" (UniqueName: \"kubernetes.io/projected/649dd6ee-b863-4784-8aff-97d27496e0c0-kube-api-access-xslrb\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6\" (UID: \"649dd6ee-b863-4784-8aff-97d27496e0c0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6"
Sep 16 18:56:13.391179 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:13.391051    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649dd6ee-b863-4784-8aff-97d27496e0c0-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6\" (UID: \"649dd6ee-b863-4784-8aff-97d27496e0c0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6"
Sep 16 18:56:13.492286 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:13.492245    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xslrb\" (UniqueName: \"kubernetes.io/projected/649dd6ee-b863-4784-8aff-97d27496e0c0-kube-api-access-xslrb\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6\" (UID: \"649dd6ee-b863-4784-8aff-97d27496e0c0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6"
Sep 16 18:56:13.492521 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:13.492331    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649dd6ee-b863-4784-8aff-97d27496e0c0-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6\" (UID: \"649dd6ee-b863-4784-8aff-97d27496e0c0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6"
Sep 16 18:56:13.494949 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:13.494922    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649dd6ee-b863-4784-8aff-97d27496e0c0-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6\" (UID: \"649dd6ee-b863-4784-8aff-97d27496e0c0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6"
Sep 16 18:56:13.500372 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:13.500317    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xslrb\" (UniqueName: \"kubernetes.io/projected/649dd6ee-b863-4784-8aff-97d27496e0c0-kube-api-access-xslrb\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6\" (UID: \"649dd6ee-b863-4784-8aff-97d27496e0c0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6"
Sep 16 18:56:13.525274 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:13.525240    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6"
Sep 16 18:56:13.806256 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:13.806223    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6"]
Sep 16 18:56:13.807989 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:56:13.807959    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649dd6ee_b863_4784_8aff_97d27496e0c0.slice/crio-4ce6ec23bc3d2f8d8db74d7acbaea9ed2d6210f5f0cd76ad7768c6afbe0dfb2f WatchSource:0}: Error finding container 4ce6ec23bc3d2f8d8db74d7acbaea9ed2d6210f5f0cd76ad7768c6afbe0dfb2f: Status 404 returned error can't find the container with id 4ce6ec23bc3d2f8d8db74d7acbaea9ed2d6210f5f0cd76ad7768c6afbe0dfb2f
Sep 16 18:56:14.549961 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:14.549924    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6" event={"ID":"649dd6ee-b863-4784-8aff-97d27496e0c0","Type":"ContainerStarted","Data":"f84d904b731b3f7b81f5a45a5043af642fab3dea2dd089b1737087c1f1d67c8f"}
Sep 16 18:56:14.550389 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:14.549978    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6" event={"ID":"649dd6ee-b863-4784-8aff-97d27496e0c0","Type":"ContainerStarted","Data":"4ce6ec23bc3d2f8d8db74d7acbaea9ed2d6210f5f0cd76ad7768c6afbe0dfb2f"}
Sep 16 18:56:14.550389 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:14.550027    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6"
Sep 16 18:56:14.594731 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:14.594671    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6" podStartSLOduration=1.594654154 podStartE2EDuration="1.594654154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:14.593642335 +0000 UTC m=+1012.953160330" watchObservedRunningTime="2026-09-16 18:56:14.594654154 +0000 UTC m=+1012.954172149"
Sep 16 18:56:45.561213 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:56:45.561181    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6"
Sep 16 18:57:00.845358 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:00.845307    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq"]
Sep 16 18:57:07.138831 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:07.138785    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq" podUID="6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:57:15.865631 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:15.865545    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq" podUID="6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6" containerName="main" containerID="cri-o://11c104794396eba438ddf23c01f368c22aa1fca9200c52654a4b823067ebf2a1" gracePeriod=45
Sep 16 18:57:17.139062 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:17.139020    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq" podUID="6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:57:27.138878 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:27.138836    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq" podUID="6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:57:27.139341 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:27.138964    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq"
Sep 16 18:57:37.138787 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:37.138742    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq" podUID="6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:57:46.089214 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:46.089172    2565 generic.go:356] "Generic (PLEG): container finished" podID="6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6" containerID="11c104794396eba438ddf23c01f368c22aa1fca9200c52654a4b823067ebf2a1" exitCode=0
Sep 16 18:57:46.089648 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:46.089247    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq" event={"ID":"6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6","Type":"ContainerDied","Data":"11c104794396eba438ddf23c01f368c22aa1fca9200c52654a4b823067ebf2a1"}
Sep 16 18:57:46.122646 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:46.122624    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq"
Sep 16 18:57:46.227811 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:46.227783    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6-tls-certs\") pod \"6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6\" (UID: \"6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6\") "
Sep 16 18:57:46.227953 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:46.227905    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpbff\" (UniqueName: \"kubernetes.io/projected/6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6-kube-api-access-vpbff\") pod \"6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6\" (UID: \"6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6\") "
Sep 16 18:57:46.229997 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:46.229968    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6" (UID: "6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:57:46.230105 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:46.230077    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6-kube-api-access-vpbff" (OuterVolumeSpecName: "kube-api-access-vpbff") pod "6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6" (UID: "6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6"). InnerVolumeSpecName "kube-api-access-vpbff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:57:46.328749 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:46.328721    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpbff\" (UniqueName: \"kubernetes.io/projected/6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6-kube-api-access-vpbff\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:57:46.328749 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:46.328747    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:57:47.096258 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:47.096228    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq"
Sep 16 18:57:47.096684 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:47.096268    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq" event={"ID":"6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6","Type":"ContainerDied","Data":"106d3ff4096c0771ae580ca0aa35873787105f848eb413a8c7b96e2fffbab709"}
Sep 16 18:57:47.096684 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:47.096335    2565 scope.go:117] "RemoveContainer" containerID="11c104794396eba438ddf23c01f368c22aa1fca9200c52654a4b823067ebf2a1"
Sep 16 18:57:47.133931 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:47.133906    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq"]
Sep 16 18:57:47.145566 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:47.145543    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecnpwq"]
Sep 16 18:57:48.207700 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:57:48.207668    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6" path="/var/lib/kubelet/pods/6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6/volumes"
Sep 16 18:58:01.905812 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:01.905779    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj"]
Sep 16 18:58:01.906222 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:01.906187    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6" containerName="main"
Sep 16 18:58:01.906222 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:01.906199    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6" containerName="main"
Sep 16 18:58:01.906310 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:01.906257    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e1e5d9b-5642-4f5d-b459-d1965d7f8ca6" containerName="main"
Sep 16 18:58:01.909447 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:01.909421    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj"
Sep 16 18:58:01.912999 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:01.912972    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 16 18:58:01.913140 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:01.912975    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 16 18:58:01.913140 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:01.913087    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-r7mcr\""
Sep 16 18:58:01.913305 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:01.913278    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 16 18:58:01.913450 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:01.913320    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-pmxtc\""
Sep 16 18:58:01.924642 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:01.924620    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj"]
Sep 16 18:58:02.066245 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:02.066212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhjk8\" (UniqueName: \"kubernetes.io/projected/46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901-kube-api-access-dhjk8\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj\" (UID: \"46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj"
Sep 16 18:58:02.066427 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:02.066270    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj\" (UID: \"46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj"
Sep 16 18:58:02.167615 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:02.167534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhjk8\" (UniqueName: \"kubernetes.io/projected/46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901-kube-api-access-dhjk8\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj\" (UID: \"46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj"
Sep 16 18:58:02.167615 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:02.167592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj\" (UID: \"46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj"
Sep 16 18:58:02.169988 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:02.169966    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj\" (UID: \"46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj"
Sep 16 18:58:02.177201 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:02.177176    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhjk8\" (UniqueName: \"kubernetes.io/projected/46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901-kube-api-access-dhjk8\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj\" (UID: \"46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj"
Sep 16 18:58:02.220718 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:02.220694    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj"
Sep 16 18:58:02.496397 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:02.496368    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj"]
Sep 16 18:58:02.499562 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:58:02.499537    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b8a48b_e3bc_4b0c_8bd7_4529b0fbe901.slice/crio-395512d44d25b5b834adda8242e7a1a8b5fec94ae31d4d0a044a65ab020d12dc WatchSource:0}: Error finding container 395512d44d25b5b834adda8242e7a1a8b5fec94ae31d4d0a044a65ab020d12dc: Status 404 returned error can't find the container with id 395512d44d25b5b834adda8242e7a1a8b5fec94ae31d4d0a044a65ab020d12dc
Sep 16 18:58:03.197892 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:03.197855    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj" event={"ID":"46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901","Type":"ContainerStarted","Data":"e7576485e7a7ca37a7b604dc71d5865b4f85d21e46a0314a56b5d433654dad53"}
Sep 16 18:58:03.198267 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:03.197940    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj" event={"ID":"46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901","Type":"ContainerStarted","Data":"395512d44d25b5b834adda8242e7a1a8b5fec94ae31d4d0a044a65ab020d12dc"}
Sep 16 18:58:03.198267 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:03.198010    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj"
Sep 16 18:58:03.222241 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:03.222185    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj" podStartSLOduration=2.222167626 podStartE2EDuration="2.222167626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:03.222120106 +0000 UTC m=+1121.581638102" watchObservedRunningTime="2026-09-16 18:58:03.222167626 +0000 UTC m=+1121.581685669"
Sep 16 18:58:18.715384 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:18.715333    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6"]
Sep 16 18:58:25.561493 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:25.561454    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6" podUID="649dd6ee-b863-4784-8aff-97d27496e0c0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:58:33.736289 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:33.736242    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6" podUID="649dd6ee-b863-4784-8aff-97d27496e0c0" containerName="main" containerID="cri-o://f84d904b731b3f7b81f5a45a5043af642fab3dea2dd089b1737087c1f1d67c8f" gracePeriod=45
Sep 16 18:58:34.208077 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:34.207999    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj"
Sep 16 18:58:35.560799 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:35.560755    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6" podUID="649dd6ee-b863-4784-8aff-97d27496e0c0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:58:36.292674 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:36.292639    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj"]
Sep 16 18:58:44.205830 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:44.205743    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj" podUID="46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:58:45.561516 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:45.561476    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6" podUID="649dd6ee-b863-4784-8aff-97d27496e0c0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:58:45.561884 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:45.561565    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6"
Sep 16 18:58:51.312172 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:51.312129    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj" podUID="46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901" containerName="main" containerID="cri-o://e7576485e7a7ca37a7b604dc71d5865b4f85d21e46a0314a56b5d433654dad53" gracePeriod=45
Sep 16 18:58:54.205953 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:54.205910    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj" podUID="46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:58:55.560920 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:58:55.560882    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6" podUID="649dd6ee-b863-4784-8aff-97d27496e0c0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:59:03.985747 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:03.985712    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6"
Sep 16 18:59:04.157710 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:04.157631    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xslrb\" (UniqueName: \"kubernetes.io/projected/649dd6ee-b863-4784-8aff-97d27496e0c0-kube-api-access-xslrb\") pod \"649dd6ee-b863-4784-8aff-97d27496e0c0\" (UID: \"649dd6ee-b863-4784-8aff-97d27496e0c0\") "
Sep 16 18:59:04.157710 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:04.157676    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649dd6ee-b863-4784-8aff-97d27496e0c0-tls-certs\") pod \"649dd6ee-b863-4784-8aff-97d27496e0c0\" (UID: \"649dd6ee-b863-4784-8aff-97d27496e0c0\") "
Sep 16 18:59:04.159870 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:04.159846    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/649dd6ee-b863-4784-8aff-97d27496e0c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "649dd6ee-b863-4784-8aff-97d27496e0c0" (UID: "649dd6ee-b863-4784-8aff-97d27496e0c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:04.160000 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:04.159979    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/649dd6ee-b863-4784-8aff-97d27496e0c0-kube-api-access-xslrb" (OuterVolumeSpecName: "kube-api-access-xslrb") pod "649dd6ee-b863-4784-8aff-97d27496e0c0" (UID: "649dd6ee-b863-4784-8aff-97d27496e0c0"). InnerVolumeSpecName "kube-api-access-xslrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:04.205727 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:04.205697    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj" podUID="46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:59:04.207375 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:04.207356    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj"
Sep 16 18:59:04.258707 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:04.258681    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xslrb\" (UniqueName: \"kubernetes.io/projected/649dd6ee-b863-4784-8aff-97d27496e0c0-kube-api-access-xslrb\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:59:04.258707 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:04.258705    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649dd6ee-b863-4784-8aff-97d27496e0c0-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:59:04.532398 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:04.532367    2565 generic.go:356] "Generic (PLEG): container finished" podID="649dd6ee-b863-4784-8aff-97d27496e0c0" containerID="f84d904b731b3f7b81f5a45a5043af642fab3dea2dd089b1737087c1f1d67c8f" exitCode=0
Sep 16 18:59:04.532564 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:04.532404    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6"
Sep 16 18:59:04.532564 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:04.532427    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6" event={"ID":"649dd6ee-b863-4784-8aff-97d27496e0c0","Type":"ContainerDied","Data":"f84d904b731b3f7b81f5a45a5043af642fab3dea2dd089b1737087c1f1d67c8f"}
Sep 16 18:59:04.532564 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:04.532478    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6" event={"ID":"649dd6ee-b863-4784-8aff-97d27496e0c0","Type":"ContainerDied","Data":"4ce6ec23bc3d2f8d8db74d7acbaea9ed2d6210f5f0cd76ad7768c6afbe0dfb2f"}
Sep 16 18:59:04.532564 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:04.532498    2565 scope.go:117] "RemoveContainer" containerID="f84d904b731b3f7b81f5a45a5043af642fab3dea2dd089b1737087c1f1d67c8f"
Sep 16 18:59:04.541478 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:04.541460    2565 scope.go:117] "RemoveContainer" containerID="f84d904b731b3f7b81f5a45a5043af642fab3dea2dd089b1737087c1f1d67c8f"
Sep 16 18:59:04.541758 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:59:04.541737    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f84d904b731b3f7b81f5a45a5043af642fab3dea2dd089b1737087c1f1d67c8f\": container with ID starting with f84d904b731b3f7b81f5a45a5043af642fab3dea2dd089b1737087c1f1d67c8f not found: ID does not exist" containerID="f84d904b731b3f7b81f5a45a5043af642fab3dea2dd089b1737087c1f1d67c8f"
Sep 16 18:59:04.541802 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:04.541764    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f84d904b731b3f7b81f5a45a5043af642fab3dea2dd089b1737087c1f1d67c8f"} err="failed to get container status \"f84d904b731b3f7b81f5a45a5043af642fab3dea2dd089b1737087c1f1d67c8f\": rpc error: code = NotFound desc = could not find container \"f84d904b731b3f7b81f5a45a5043af642fab3dea2dd089b1737087c1f1d67c8f\": container with ID starting with f84d904b731b3f7b81f5a45a5043af642fab3dea2dd089b1737087c1f1d67c8f not found: ID does not exist"
Sep 16 18:59:04.556664 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:04.556640    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6"]
Sep 16 18:59:04.563956 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:04.563937    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-nkdw6"]
Sep 16 18:59:06.207470 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:06.207437    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="649dd6ee-b863-4784-8aff-97d27496e0c0" path="/var/lib/kubelet/pods/649dd6ee-b863-4784-8aff-97d27496e0c0/volumes"
Sep 16 18:59:14.205919 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:14.205879    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj" podUID="46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:59:21.558852 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:21.558827    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj"
Sep 16 18:59:21.599431 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:21.599373    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901-tls-certs\") pod \"46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901\" (UID: \"46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901\") "
Sep 16 18:59:21.599431 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:21.599414    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhjk8\" (UniqueName: \"kubernetes.io/projected/46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901-kube-api-access-dhjk8\") pod \"46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901\" (UID: \"46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901\") "
Sep 16 18:59:21.601573 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:21.601540    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901-kube-api-access-dhjk8" (OuterVolumeSpecName: "kube-api-access-dhjk8") pod "46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901" (UID: "46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901"). InnerVolumeSpecName "kube-api-access-dhjk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:21.601573 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:21.601562    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901" (UID: "46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:21.627572 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:21.627547    2565 generic.go:356] "Generic (PLEG): container finished" podID="46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901" containerID="e7576485e7a7ca37a7b604dc71d5865b4f85d21e46a0314a56b5d433654dad53" exitCode=0
Sep 16 18:59:21.627672 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:21.627589    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj"
Sep 16 18:59:21.627672 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:21.627635    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj" event={"ID":"46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901","Type":"ContainerDied","Data":"e7576485e7a7ca37a7b604dc71d5865b4f85d21e46a0314a56b5d433654dad53"}
Sep 16 18:59:21.627791 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:21.627695    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj" event={"ID":"46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901","Type":"ContainerDied","Data":"395512d44d25b5b834adda8242e7a1a8b5fec94ae31d4d0a044a65ab020d12dc"}
Sep 16 18:59:21.627791 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:21.627727    2565 scope.go:117] "RemoveContainer" containerID="e7576485e7a7ca37a7b604dc71d5865b4f85d21e46a0314a56b5d433654dad53"
Sep 16 18:59:21.636877 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:21.636861    2565 scope.go:117] "RemoveContainer" containerID="e7576485e7a7ca37a7b604dc71d5865b4f85d21e46a0314a56b5d433654dad53"
Sep 16 18:59:21.637121 ip-10-0-130-204 kubenswrapper[2565]: E0916 18:59:21.637102    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7576485e7a7ca37a7b604dc71d5865b4f85d21e46a0314a56b5d433654dad53\": container with ID starting with e7576485e7a7ca37a7b604dc71d5865b4f85d21e46a0314a56b5d433654dad53 not found: ID does not exist" containerID="e7576485e7a7ca37a7b604dc71d5865b4f85d21e46a0314a56b5d433654dad53"
Sep 16 18:59:21.637187 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:21.637134    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7576485e7a7ca37a7b604dc71d5865b4f85d21e46a0314a56b5d433654dad53"} err="failed to get container status \"e7576485e7a7ca37a7b604dc71d5865b4f85d21e46a0314a56b5d433654dad53\": rpc error: code = NotFound desc = could not find container \"e7576485e7a7ca37a7b604dc71d5865b4f85d21e46a0314a56b5d433654dad53\": container with ID starting with e7576485e7a7ca37a7b604dc71d5865b4f85d21e46a0314a56b5d433654dad53 not found: ID does not exist"
Sep 16 18:59:21.655417 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:21.655396    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj"]
Sep 16 18:59:21.660333 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:21.660314    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbgmjlj"]
Sep 16 18:59:21.700612 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:21.700590    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:59:21.700612 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:21.700610    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhjk8\" (UniqueName: \"kubernetes.io/projected/46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901-kube-api-access-dhjk8\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 18:59:22.208865 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:22.208832    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901" path="/var/lib/kubelet/pods/46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901/volumes"
Sep 16 18:59:34.097987 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.097952    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"]
Sep 16 18:59:34.098340 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.098325    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901" containerName="main"
Sep 16 18:59:34.098340 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.098336    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901" containerName="main"
Sep 16 18:59:34.098444 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.098391    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="649dd6ee-b863-4784-8aff-97d27496e0c0" containerName="main"
Sep 16 18:59:34.098444 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.098398    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="649dd6ee-b863-4784-8aff-97d27496e0c0" containerName="main"
Sep 16 18:59:34.098511 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.098453    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="649dd6ee-b863-4784-8aff-97d27496e0c0" containerName="main"
Sep 16 18:59:34.098511 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.098464    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b8a48b-e3bc-4b0c-8bd7-4529b0fbe901" containerName="main"
Sep 16 18:59:34.102960 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.102937    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.108760 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.108737    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-dktdn\""
Sep 16 18:59:34.108760 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.108756    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 16 18:59:34.108950 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.108870    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 16 18:59:34.109008 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.108966    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 16 18:59:34.122571 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.122550    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"]
Sep 16 18:59:34.194480 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.194455    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.194593 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.194488    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.194593 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.194520    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.194676 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.194621    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.194712 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.194675    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.194750 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.194723    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzwg9\" (UniqueName: \"kubernetes.io/projected/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-kube-api-access-jzwg9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.295938 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.295900    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.296069 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.295961    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.296069 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.296000    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzwg9\" (UniqueName: \"kubernetes.io/projected/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-kube-api-access-jzwg9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.296069 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.296031    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.296069 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.296055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.296245 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.296085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.296454 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.296427    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.296454 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.296444    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.296603 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.296482    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.298109 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.298089    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.298415 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.298396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.307715 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.307697    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzwg9\" (UniqueName: \"kubernetes.io/projected/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-kube-api-access-jzwg9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.413542 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.413476    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:34.698147 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:34.698120    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"]
Sep 16 18:59:34.700264 ip-10-0-130-204 kubenswrapper[2565]: W0916 18:59:34.700240    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6bc794f_ee6b_4b70_9cb8_1e4aa55b77ad.slice/crio-006bd15cf7d5f036a7cb2f56a25d9416dd9e1451b1fdc7ebcf5b2f40ff73ca57 WatchSource:0}: Error finding container 006bd15cf7d5f036a7cb2f56a25d9416dd9e1451b1fdc7ebcf5b2f40ff73ca57: Status 404 returned error can't find the container with id 006bd15cf7d5f036a7cb2f56a25d9416dd9e1451b1fdc7ebcf5b2f40ff73ca57
Sep 16 18:59:35.700331 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:35.700292    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr" event={"ID":"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad","Type":"ContainerStarted","Data":"ebfeb26cdc840dce0d52d1ed1636cf89fdeac1cd58270094b541e8e7e9a6abcd"}
Sep 16 18:59:35.700727 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:35.700376    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr" event={"ID":"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad","Type":"ContainerStarted","Data":"006bd15cf7d5f036a7cb2f56a25d9416dd9e1451b1fdc7ebcf5b2f40ff73ca57"}
Sep 16 18:59:35.750586 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:35.750535    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr" podStartSLOduration=1.7505213560000001 podStartE2EDuration="1.750521356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:35.749739551 +0000 UTC m=+1214.109257549" watchObservedRunningTime="2026-09-16 18:59:35.750521356 +0000 UTC m=+1214.110039415"
Sep 16 18:59:36.414467 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:36.414432    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:44.414023 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:44.413986    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:44.426953 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:44.426922    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 18:59:44.757509 ip-10-0-130-204 kubenswrapper[2565]: I0916 18:59:44.757480    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 19:00:08.366357 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.366311    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"]
Sep 16 19:00:08.375229 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:00:08.375184    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:00:08.375229 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:00:08.375229 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 19:00:08.375229 ip-10-0-130-204 kubenswrapper[2565]:  > containerID="ebfeb26cdc840dce0d52d1ed1636cf89fdeac1cd58270094b541e8e7e9a6abcd" cmd=["/bin/sleep","15"]
Sep 16 19:00:08.375485 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.375238    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr" message="" err=<
Sep 16 19:00:08.375485 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:00:08.375485 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 19:00:08.375485 ip-10-0-130-204 kubenswrapper[2565]:  >
Sep 16 19:00:08.375485 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:00:08.375250    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:00:08.375485 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:00:08.375485 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 19:00:08.375485 ip-10-0-130-204 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr" podUID="f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad" containerName="main" containerID="cri-o://ebfeb26cdc840dce0d52d1ed1636cf89fdeac1cd58270094b541e8e7e9a6abcd"
Sep 16 19:00:08.375485 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.375300    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr" podUID="f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad" containerName="main" containerID="cri-o://ebfeb26cdc840dce0d52d1ed1636cf89fdeac1cd58270094b541e8e7e9a6abcd" gracePeriod=60
Sep 16 19:00:08.731922 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.731897    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 19:00:08.861370 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.861331    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-home\") pod \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") "
Sep 16 19:00:08.861512 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.861379    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-tls-certs\") pod \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") "
Sep 16 19:00:08.861512 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.861408    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-dshm\") pod \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") "
Sep 16 19:00:08.861512 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.861428    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-tmp-dir\") pod \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") "
Sep 16 19:00:08.861512 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.861455    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-model-cache\") pod \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") "
Sep 16 19:00:08.861512 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.861475    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzwg9\" (UniqueName: \"kubernetes.io/projected/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-kube-api-access-jzwg9\") pod \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\" (UID: \"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad\") "
Sep 16 19:00:08.861764 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.861611    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-home" (OuterVolumeSpecName: "home") pod "f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad" (UID: "f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:08.861764 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.861699    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad" (UID: "f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:08.861764 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.861719    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad" (UID: "f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:08.861764 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.861738    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:00:08.863681 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.863656    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-dshm" (OuterVolumeSpecName: "dshm") pod "f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad" (UID: "f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:08.863805 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.863687    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad" (UID: "f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:08.863997 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.863980    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-kube-api-access-jzwg9" (OuterVolumeSpecName: "kube-api-access-jzwg9") pod "f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad" (UID: "f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad"). InnerVolumeSpecName "kube-api-access-jzwg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:08.877243 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.877222    2565 generic.go:356] "Generic (PLEG): container finished" podID="f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad" containerID="ebfeb26cdc840dce0d52d1ed1636cf89fdeac1cd58270094b541e8e7e9a6abcd" exitCode=0
Sep 16 19:00:08.877356 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.877262    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"
Sep 16 19:00:08.877356 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.877307    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr" event={"ID":"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad","Type":"ContainerDied","Data":"ebfeb26cdc840dce0d52d1ed1636cf89fdeac1cd58270094b541e8e7e9a6abcd"}
Sep 16 19:00:08.877453 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.877386    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr" event={"ID":"f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad","Type":"ContainerDied","Data":"006bd15cf7d5f036a7cb2f56a25d9416dd9e1451b1fdc7ebcf5b2f40ff73ca57"}
Sep 16 19:00:08.877453 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.877410    2565 scope.go:117] "RemoveContainer" containerID="ebfeb26cdc840dce0d52d1ed1636cf89fdeac1cd58270094b541e8e7e9a6abcd"
Sep 16 19:00:08.887131 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.887109    2565 scope.go:117] "RemoveContainer" containerID="ebfeb26cdc840dce0d52d1ed1636cf89fdeac1cd58270094b541e8e7e9a6abcd"
Sep 16 19:00:08.887423 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:00:08.887397    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebfeb26cdc840dce0d52d1ed1636cf89fdeac1cd58270094b541e8e7e9a6abcd\": container with ID starting with ebfeb26cdc840dce0d52d1ed1636cf89fdeac1cd58270094b541e8e7e9a6abcd not found: ID does not exist" containerID="ebfeb26cdc840dce0d52d1ed1636cf89fdeac1cd58270094b541e8e7e9a6abcd"
Sep 16 19:00:08.887520 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.887431    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfeb26cdc840dce0d52d1ed1636cf89fdeac1cd58270094b541e8e7e9a6abcd"} err="failed to get container status \"ebfeb26cdc840dce0d52d1ed1636cf89fdeac1cd58270094b541e8e7e9a6abcd\": rpc error: code = NotFound desc = could not find container \"ebfeb26cdc840dce0d52d1ed1636cf89fdeac1cd58270094b541e8e7e9a6abcd\": container with ID starting with ebfeb26cdc840dce0d52d1ed1636cf89fdeac1cd58270094b541e8e7e9a6abcd not found: ID does not exist"
Sep 16 19:00:08.956942 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.956880    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"]
Sep 16 19:00:08.962923 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.962904    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:00:08.962988 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.962925    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:00:08.962988 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.962933    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:00:08.962988 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.962941    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:00:08.962988 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.962950    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzwg9\" (UniqueName: \"kubernetes.io/projected/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad-kube-api-access-jzwg9\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:00:08.972988 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:08.972961    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fljdkr"]
Sep 16 19:00:10.207099 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:00:10.207064    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad" path="/var/lib/kubelet/pods/f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad/volumes"
Sep 16 19:01:01.322111 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.322076    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"]
Sep 16 19:01:01.322531 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.322521    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad" containerName="main"
Sep 16 19:01:01.322577 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.322537    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad" containerName="main"
Sep 16 19:01:01.322624 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.322608    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6bc794f-ee6b-4b70-9cb8-1e4aa55b77ad" containerName="main"
Sep 16 19:01:01.325264 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.325247    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.329642 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.329613    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-6l6g9\""
Sep 16 19:01:01.329760 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.329620    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 16 19:01:01.329992 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.329977    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 16 19:01:01.341694 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.341674    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 16 19:01:01.361033 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.361007    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"]
Sep 16 19:01:01.454474 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.454449    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.454601 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.454486    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxwmk\" (UniqueName: \"kubernetes.io/projected/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-kube-api-access-fxwmk\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.454601 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.454512    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.454601 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.454579    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.454715 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.454620    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.454715 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.454655    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.555191 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.555167    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.555311 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.555196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.555311 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.555220    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.555311 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.555245    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.555311 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.555280    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.555311 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.555310    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxwmk\" (UniqueName: \"kubernetes.io/projected/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-kube-api-access-fxwmk\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.555604 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.555556    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.555664 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.555610    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.555725 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.555703    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.557632 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.557609    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.557770 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.557750    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.566283 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.566256    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxwmk\" (UniqueName: \"kubernetes.io/projected/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-kube-api-access-fxwmk\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.635784 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.635718    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:01.930651 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:01.930598    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"]
Sep 16 19:01:01.933514 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:01:01.933477    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358d700b_4c20_4ea7_9d1d_3e1ff07003dc.slice/crio-dcb4047164e957144f22a6524df3daa9b0a0cc8c1d148f0d7c267faba7ba798f WatchSource:0}: Error finding container dcb4047164e957144f22a6524df3daa9b0a0cc8c1d148f0d7c267faba7ba798f: Status 404 returned error can't find the container with id dcb4047164e957144f22a6524df3daa9b0a0cc8c1d148f0d7c267faba7ba798f
Sep 16 19:01:02.164003 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:02.163971    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs" event={"ID":"358d700b-4c20-4ea7-9d1d-3e1ff07003dc","Type":"ContainerStarted","Data":"593acfbd4398477c93cc1162765da3a66dc78cda2d5ce97ebd207e9c39c7e4ae"}
Sep 16 19:01:02.164142 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:02.164023    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs" event={"ID":"358d700b-4c20-4ea7-9d1d-3e1ff07003dc","Type":"ContainerStarted","Data":"dcb4047164e957144f22a6524df3daa9b0a0cc8c1d148f0d7c267faba7ba798f"}
Sep 16 19:01:02.193203 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:02.193117    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs" podStartSLOduration=1.193103269 podStartE2EDuration="1.193103269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:02.191605404 +0000 UTC m=+1300.551123399" watchObservedRunningTime="2026-09-16 19:01:02.193103269 +0000 UTC m=+1300.552621305"
Sep 16 19:01:02.635814 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:02.635783    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:11.636062 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:11.636022    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:11.648096 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:11.648069    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:12.231623 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:12.231597    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:35.632812 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:35.632759    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"]
Sep 16 19:01:35.643117 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:01:35.643069    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:01:35.643117 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:01:35.643117 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 19:01:35.643117 ip-10-0-130-204 kubenswrapper[2565]:  > containerID="593acfbd4398477c93cc1162765da3a66dc78cda2d5ce97ebd207e9c39c7e4ae" cmd=["/bin/sleep","15"]
Sep 16 19:01:35.643485 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:35.643133    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs" message="" err=<
Sep 16 19:01:35.643485 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:01:35.643485 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 19:01:35.643485 ip-10-0-130-204 kubenswrapper[2565]:  >
Sep 16 19:01:35.643485 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:01:35.643149    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:01:35.643485 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:01:35.643485 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 19:01:35.643485 ip-10-0-130-204 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs" podUID="358d700b-4c20-4ea7-9d1d-3e1ff07003dc" containerName="main" containerID="cri-o://593acfbd4398477c93cc1162765da3a66dc78cda2d5ce97ebd207e9c39c7e4ae"
Sep 16 19:01:35.643485 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:35.643188    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs" podUID="358d700b-4c20-4ea7-9d1d-3e1ff07003dc" containerName="main" containerID="cri-o://593acfbd4398477c93cc1162765da3a66dc78cda2d5ce97ebd207e9c39c7e4ae" gracePeriod=60
Sep 16 19:01:36.003340 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.003316    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:36.122880 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.122849    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-tls-certs\") pod \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") "
Sep 16 19:01:36.122880 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.122878    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-dshm\") pod \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") "
Sep 16 19:01:36.123044 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.122922    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-tmp-dir\") pod \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") "
Sep 16 19:01:36.123090 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.123054    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-model-cache\") pod \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") "
Sep 16 19:01:36.123180 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.123159    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "358d700b-4c20-4ea7-9d1d-3e1ff07003dc" (UID: "358d700b-4c20-4ea7-9d1d-3e1ff07003dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:36.123232 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.123169    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-home\") pod \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") "
Sep 16 19:01:36.123291 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.123228    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxwmk\" (UniqueName: \"kubernetes.io/projected/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-kube-api-access-fxwmk\") pod \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\" (UID: \"358d700b-4c20-4ea7-9d1d-3e1ff07003dc\") "
Sep 16 19:01:36.123366 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.123315    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "358d700b-4c20-4ea7-9d1d-3e1ff07003dc" (UID: "358d700b-4c20-4ea7-9d1d-3e1ff07003dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:36.123366 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.123327    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-home" (OuterVolumeSpecName: "home") pod "358d700b-4c20-4ea7-9d1d-3e1ff07003dc" (UID: "358d700b-4c20-4ea7-9d1d-3e1ff07003dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:36.123526 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.123507    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:01:36.123571 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.123529    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:01:36.123571 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.123537    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:01:36.125132 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.125111    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-dshm" (OuterVolumeSpecName: "dshm") pod "358d700b-4c20-4ea7-9d1d-3e1ff07003dc" (UID: "358d700b-4c20-4ea7-9d1d-3e1ff07003dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:36.125216 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.125153    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "358d700b-4c20-4ea7-9d1d-3e1ff07003dc" (UID: "358d700b-4c20-4ea7-9d1d-3e1ff07003dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:36.125295 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.125273    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-kube-api-access-fxwmk" (OuterVolumeSpecName: "kube-api-access-fxwmk") pod "358d700b-4c20-4ea7-9d1d-3e1ff07003dc" (UID: "358d700b-4c20-4ea7-9d1d-3e1ff07003dc"). InnerVolumeSpecName "kube-api-access-fxwmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:36.224677 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.224656    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxwmk\" (UniqueName: \"kubernetes.io/projected/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-kube-api-access-fxwmk\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:01:36.224677 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.224678    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:01:36.224817 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.224687    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358d700b-4c20-4ea7-9d1d-3e1ff07003dc-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:01:36.350596 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.350569    2565 generic.go:356] "Generic (PLEG): container finished" podID="358d700b-4c20-4ea7-9d1d-3e1ff07003dc" containerID="593acfbd4398477c93cc1162765da3a66dc78cda2d5ce97ebd207e9c39c7e4ae" exitCode=0
Sep 16 19:01:36.350713 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.350604    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"
Sep 16 19:01:36.350713 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.350692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs" event={"ID":"358d700b-4c20-4ea7-9d1d-3e1ff07003dc","Type":"ContainerDied","Data":"593acfbd4398477c93cc1162765da3a66dc78cda2d5ce97ebd207e9c39c7e4ae"}
Sep 16 19:01:36.350833 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.350733    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs" event={"ID":"358d700b-4c20-4ea7-9d1d-3e1ff07003dc","Type":"ContainerDied","Data":"dcb4047164e957144f22a6524df3daa9b0a0cc8c1d148f0d7c267faba7ba798f"}
Sep 16 19:01:36.350833 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.350754    2565 scope.go:117] "RemoveContainer" containerID="593acfbd4398477c93cc1162765da3a66dc78cda2d5ce97ebd207e9c39c7e4ae"
Sep 16 19:01:36.362498 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.362478    2565 scope.go:117] "RemoveContainer" containerID="593acfbd4398477c93cc1162765da3a66dc78cda2d5ce97ebd207e9c39c7e4ae"
Sep 16 19:01:36.362785 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:01:36.362765    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"593acfbd4398477c93cc1162765da3a66dc78cda2d5ce97ebd207e9c39c7e4ae\": container with ID starting with 593acfbd4398477c93cc1162765da3a66dc78cda2d5ce97ebd207e9c39c7e4ae not found: ID does not exist" containerID="593acfbd4398477c93cc1162765da3a66dc78cda2d5ce97ebd207e9c39c7e4ae"
Sep 16 19:01:36.362847 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.362792    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"593acfbd4398477c93cc1162765da3a66dc78cda2d5ce97ebd207e9c39c7e4ae"} err="failed to get container status \"593acfbd4398477c93cc1162765da3a66dc78cda2d5ce97ebd207e9c39c7e4ae\": rpc error: code = NotFound desc = could not find container \"593acfbd4398477c93cc1162765da3a66dc78cda2d5ce97ebd207e9c39c7e4ae\": container with ID starting with 593acfbd4398477c93cc1162765da3a66dc78cda2d5ce97ebd207e9c39c7e4ae not found: ID does not exist"
Sep 16 19:01:36.405904 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.405880    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"]
Sep 16 19:01:36.408798 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:36.408777    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-b5kfs"]
Sep 16 19:01:38.207685 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:01:38.207648    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="358d700b-4c20-4ea7-9d1d-3e1ff07003dc" path="/var/lib/kubelet/pods/358d700b-4c20-4ea7-9d1d-3e1ff07003dc/volumes"
Sep 16 19:02:16.728111 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:16.728073    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr"]
Sep 16 19:02:16.729031 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:16.729006    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358d700b-4c20-4ea7-9d1d-3e1ff07003dc" containerName="main"
Sep 16 19:02:16.729031 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:16.729032    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="358d700b-4c20-4ea7-9d1d-3e1ff07003dc" containerName="main"
Sep 16 19:02:16.729206 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:16.729124    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="358d700b-4c20-4ea7-9d1d-3e1ff07003dc" containerName="main"
Sep 16 19:02:16.732389 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:16.732369    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr"
Sep 16 19:02:16.735114 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:16.735090    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-5ccsq\""
Sep 16 19:02:16.735942 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:16.735916    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 16 19:02:16.736643 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:16.736621    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 16 19:02:16.737094 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:16.737071    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-s47ws\""
Sep 16 19:02:16.737094 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:16.737088    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 16 19:02:16.743754 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:16.743727    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr"]
Sep 16 19:02:16.822550 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:16.822520    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e7bceeb-edc0-46f1-aa58-2857fcc65861-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr\" (UID: \"9e7bceeb-edc0-46f1-aa58-2857fcc65861\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr"
Sep 16 19:02:16.822679 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:16.822562    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r72m\" (UniqueName: \"kubernetes.io/projected/9e7bceeb-edc0-46f1-aa58-2857fcc65861-kube-api-access-5r72m\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr\" (UID: \"9e7bceeb-edc0-46f1-aa58-2857fcc65861\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr"
Sep 16 19:02:16.923186 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:16.923156    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e7bceeb-edc0-46f1-aa58-2857fcc65861-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr\" (UID: \"9e7bceeb-edc0-46f1-aa58-2857fcc65861\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr"
Sep 16 19:02:16.923321 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:16.923200    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r72m\" (UniqueName: \"kubernetes.io/projected/9e7bceeb-edc0-46f1-aa58-2857fcc65861-kube-api-access-5r72m\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr\" (UID: \"9e7bceeb-edc0-46f1-aa58-2857fcc65861\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr"
Sep 16 19:02:16.925620 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:16.925594    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e7bceeb-edc0-46f1-aa58-2857fcc65861-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr\" (UID: \"9e7bceeb-edc0-46f1-aa58-2857fcc65861\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr"
Sep 16 19:02:16.931838 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:16.931813    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r72m\" (UniqueName: \"kubernetes.io/projected/9e7bceeb-edc0-46f1-aa58-2857fcc65861-kube-api-access-5r72m\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr\" (UID: \"9e7bceeb-edc0-46f1-aa58-2857fcc65861\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr"
Sep 16 19:02:17.044487 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:17.044428    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr"
Sep 16 19:02:17.329651 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:17.329618    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr"]
Sep 16 19:02:17.331102 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:02:17.331073    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7bceeb_edc0_46f1_aa58_2857fcc65861.slice/crio-758c552f2f8ec94aaa53531c90a4bd1975d7bd7fca561070d47b398f724e03f1 WatchSource:0}: Error finding container 758c552f2f8ec94aaa53531c90a4bd1975d7bd7fca561070d47b398f724e03f1: Status 404 returned error can't find the container with id 758c552f2f8ec94aaa53531c90a4bd1975d7bd7fca561070d47b398f724e03f1
Sep 16 19:02:17.580713 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:17.580637    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr" event={"ID":"9e7bceeb-edc0-46f1-aa58-2857fcc65861","Type":"ContainerStarted","Data":"adc333c0749910ff72c19efd5641bd306c6dce963ce3c69763316fd2ff9c741b"}
Sep 16 19:02:17.580713 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:17.580687    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr" event={"ID":"9e7bceeb-edc0-46f1-aa58-2857fcc65861","Type":"ContainerStarted","Data":"758c552f2f8ec94aaa53531c90a4bd1975d7bd7fca561070d47b398f724e03f1"}
Sep 16 19:02:17.580937 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:17.580762    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr"
Sep 16 19:02:17.611041 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:17.610996    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr" podStartSLOduration=1.6109816750000001 podStartE2EDuration="1.610981675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:17.609075737 +0000 UTC m=+1375.968593729" watchObservedRunningTime="2026-09-16 19:02:17.610981675 +0000 UTC m=+1375.970499667"
Sep 16 19:02:48.589203 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:02:48.589174    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr"
Sep 16 19:04:31.104395 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:04:31.104362    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr"]
Sep 16 19:04:38.589097 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:04:38.589058    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr" podUID="9e7bceeb-edc0-46f1-aa58-2857fcc65861" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:46.124126 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:04:46.124033    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr" podUID="9e7bceeb-edc0-46f1-aa58-2857fcc65861" containerName="main" containerID="cri-o://adc333c0749910ff72c19efd5641bd306c6dce963ce3c69763316fd2ff9c741b" gracePeriod=45
Sep 16 19:04:48.589093 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:04:48.589048    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr" podUID="9e7bceeb-edc0-46f1-aa58-2857fcc65861" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:58.588244 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:04:58.588202    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr" podUID="9e7bceeb-edc0-46f1-aa58-2857fcc65861" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:58.588650 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:04:58.588316    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr"
Sep 16 19:05:08.589176 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:08.589136    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr" podUID="9e7bceeb-edc0-46f1-aa58-2857fcc65861" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:05:16.375555 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:16.375530    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr"
Sep 16 19:05:16.479826 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:16.479801    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e7bceeb-edc0-46f1-aa58-2857fcc65861-tls-certs\") pod \"9e7bceeb-edc0-46f1-aa58-2857fcc65861\" (UID: \"9e7bceeb-edc0-46f1-aa58-2857fcc65861\") "
Sep 16 19:05:16.479961 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:16.479867    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r72m\" (UniqueName: \"kubernetes.io/projected/9e7bceeb-edc0-46f1-aa58-2857fcc65861-kube-api-access-5r72m\") pod \"9e7bceeb-edc0-46f1-aa58-2857fcc65861\" (UID: \"9e7bceeb-edc0-46f1-aa58-2857fcc65861\") "
Sep 16 19:05:16.482061 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:16.482025    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e7bceeb-edc0-46f1-aa58-2857fcc65861-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e7bceeb-edc0-46f1-aa58-2857fcc65861" (UID: "9e7bceeb-edc0-46f1-aa58-2857fcc65861"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:16.482161 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:16.482076    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7bceeb-edc0-46f1-aa58-2857fcc65861-kube-api-access-5r72m" (OuterVolumeSpecName: "kube-api-access-5r72m") pod "9e7bceeb-edc0-46f1-aa58-2857fcc65861" (UID: "9e7bceeb-edc0-46f1-aa58-2857fcc65861"). InnerVolumeSpecName "kube-api-access-5r72m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:16.562978 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:16.562945    2565 generic.go:356] "Generic (PLEG): container finished" podID="9e7bceeb-edc0-46f1-aa58-2857fcc65861" containerID="adc333c0749910ff72c19efd5641bd306c6dce963ce3c69763316fd2ff9c741b" exitCode=0
Sep 16 19:05:16.563103 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:16.563003    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr"
Sep 16 19:05:16.563103 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:16.563028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr" event={"ID":"9e7bceeb-edc0-46f1-aa58-2857fcc65861","Type":"ContainerDied","Data":"adc333c0749910ff72c19efd5641bd306c6dce963ce3c69763316fd2ff9c741b"}
Sep 16 19:05:16.563179 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:16.563103    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr" event={"ID":"9e7bceeb-edc0-46f1-aa58-2857fcc65861","Type":"ContainerDied","Data":"758c552f2f8ec94aaa53531c90a4bd1975d7bd7fca561070d47b398f724e03f1"}
Sep 16 19:05:16.563179 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:16.563124    2565 scope.go:117] "RemoveContainer" containerID="adc333c0749910ff72c19efd5641bd306c6dce963ce3c69763316fd2ff9c741b"
Sep 16 19:05:16.576309 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:16.576287    2565 scope.go:117] "RemoveContainer" containerID="adc333c0749910ff72c19efd5641bd306c6dce963ce3c69763316fd2ff9c741b"
Sep 16 19:05:16.576825 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:05:16.576799    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adc333c0749910ff72c19efd5641bd306c6dce963ce3c69763316fd2ff9c741b\": container with ID starting with adc333c0749910ff72c19efd5641bd306c6dce963ce3c69763316fd2ff9c741b not found: ID does not exist" containerID="adc333c0749910ff72c19efd5641bd306c6dce963ce3c69763316fd2ff9c741b"
Sep 16 19:05:16.576974 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:16.576830    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc333c0749910ff72c19efd5641bd306c6dce963ce3c69763316fd2ff9c741b"} err="failed to get container status \"adc333c0749910ff72c19efd5641bd306c6dce963ce3c69763316fd2ff9c741b\": rpc error: code = NotFound desc = could not find container \"adc333c0749910ff72c19efd5641bd306c6dce963ce3c69763316fd2ff9c741b\": container with ID starting with adc333c0749910ff72c19efd5641bd306c6dce963ce3c69763316fd2ff9c741b not found: ID does not exist"
Sep 16 19:05:16.581082 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:16.581064    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e7bceeb-edc0-46f1-aa58-2857fcc65861-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:05:16.581165 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:16.581084    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r72m\" (UniqueName: \"kubernetes.io/projected/9e7bceeb-edc0-46f1-aa58-2857fcc65861-kube-api-access-5r72m\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:05:16.588956 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:16.588934    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr"]
Sep 16 19:05:16.596443 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:16.596416    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-rs8kr"]
Sep 16 19:05:18.207258 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:18.207229    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e7bceeb-edc0-46f1-aa58-2857fcc65861" path="/var/lib/kubelet/pods/9e7bceeb-edc0-46f1-aa58-2857fcc65861/volumes"
Sep 16 19:05:32.968685 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:32.968649    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"]
Sep 16 19:05:32.969050 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:32.969037    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e7bceeb-edc0-46f1-aa58-2857fcc65861" containerName="main"
Sep 16 19:05:32.969104 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:32.969052    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e7bceeb-edc0-46f1-aa58-2857fcc65861" containerName="main"
Sep 16 19:05:32.969140 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:32.969122    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e7bceeb-edc0-46f1-aa58-2857fcc65861" containerName="main"
Sep 16 19:05:32.972239 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:32.972218    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:32.975437 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:32.975418    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 16 19:05:32.976251 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:32.976231    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-99bv8\""
Sep 16 19:05:32.976413 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:32.976396    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\""
Sep 16 19:05:32.976501 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:32.976400    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 16 19:05:32.984421 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:32.984400    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"]
Sep 16 19:05:33.103209 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.103181    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-hctx6\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:33.103396 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.103216    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-home\") pod \"canary-v1-kserve-65d58ccc9-hctx6\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:33.103396 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.103242    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvb4m\" (UniqueName: \"kubernetes.io/projected/e9556bd7-a21f-4537-8815-99e31c8112b1-kube-api-access-gvb4m\") pod \"canary-v1-kserve-65d58ccc9-hctx6\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:33.103396 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.103309    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9556bd7-a21f-4537-8815-99e31c8112b1-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-hctx6\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:33.103568 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.103423    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-dshm\") pod \"canary-v1-kserve-65d58ccc9-hctx6\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:33.103568 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.103458    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-model-cache\") pod \"canary-v1-kserve-65d58ccc9-hctx6\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:33.204196 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.204171    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-hctx6\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:33.204321 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.204209    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-home\") pod \"canary-v1-kserve-65d58ccc9-hctx6\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:33.204321 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.204242    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvb4m\" (UniqueName: \"kubernetes.io/projected/e9556bd7-a21f-4537-8815-99e31c8112b1-kube-api-access-gvb4m\") pod \"canary-v1-kserve-65d58ccc9-hctx6\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:33.204321 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.204275    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9556bd7-a21f-4537-8815-99e31c8112b1-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-hctx6\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:33.204321 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.204307    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-dshm\") pod \"canary-v1-kserve-65d58ccc9-hctx6\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:33.204573 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.204327    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-model-cache\") pod \"canary-v1-kserve-65d58ccc9-hctx6\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:33.204646 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.204623    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-hctx6\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:33.204704 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.204641    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-home\") pod \"canary-v1-kserve-65d58ccc9-hctx6\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:33.204787 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.204767    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-model-cache\") pod \"canary-v1-kserve-65d58ccc9-hctx6\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:33.206644 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.206623    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-dshm\") pod \"canary-v1-kserve-65d58ccc9-hctx6\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:33.206839 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.206822    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9556bd7-a21f-4537-8815-99e31c8112b1-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-hctx6\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:33.214862 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.214840    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvb4m\" (UniqueName: \"kubernetes.io/projected/e9556bd7-a21f-4537-8815-99e31c8112b1-kube-api-access-gvb4m\") pod \"canary-v1-kserve-65d58ccc9-hctx6\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:33.284074 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.284051    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:33.571857 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.571832    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"]
Sep 16 19:05:33.574158 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:05:33.574114    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9556bd7_a21f_4537_8815_99e31c8112b1.slice/crio-6cf92a064eb535a8139ed07aa4b58d416c9de388d4da057a434e9ecb3eea8722 WatchSource:0}: Error finding container 6cf92a064eb535a8139ed07aa4b58d416c9de388d4da057a434e9ecb3eea8722: Status 404 returned error can't find the container with id 6cf92a064eb535a8139ed07aa4b58d416c9de388d4da057a434e9ecb3eea8722
Sep 16 19:05:33.575874 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.575858    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:05:33.654390 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:33.654341    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6" event={"ID":"e9556bd7-a21f-4537-8815-99e31c8112b1","Type":"ContainerStarted","Data":"6cf92a064eb535a8139ed07aa4b58d416c9de388d4da057a434e9ecb3eea8722"}
Sep 16 19:05:34.661431 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:34.661404    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6" event={"ID":"e9556bd7-a21f-4537-8815-99e31c8112b1","Type":"ContainerStarted","Data":"e9535080314043f5a43e6148f6fed474e8b63703847dc0af4a1d61904f85149e"}
Sep 16 19:05:35.284401 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:35.284364    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:43.285100 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:43.285058    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:43.297805 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:43.297775    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:43.318941 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:43.318894    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6" podStartSLOduration=10.335199164 podStartE2EDuration="11.318879929s" totalImagesPullingTime="983.680765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:34.680780935 +0000 UTC m=+1573.040298911" watchObservedRunningTime="2026-09-16 19:05:43.318879929 +0000 UTC m=+1581.678397954"
Sep 16 19:05:43.721176 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:43.721098    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:05:50.064747 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.064710    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"]
Sep 16 19:05:50.068408 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.068388    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.071222 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.071204    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\""
Sep 16 19:05:50.084316 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.084291    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"]
Sep 16 19:05:50.234637 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.234606    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-home\") pod \"canary-v2-kserve-7f65dbbccc-p9jch\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.234637 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.234638    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s9f2\" (UniqueName: \"kubernetes.io/projected/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-kube-api-access-8s9f2\") pod \"canary-v2-kserve-7f65dbbccc-p9jch\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.234860 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.234682    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-p9jch\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.234860 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.234710    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-p9jch\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.234860 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.234747    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-dshm\") pod \"canary-v2-kserve-7f65dbbccc-p9jch\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.234860 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.234819    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-p9jch\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.335522 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.335443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-p9jch\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.335522 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.335488    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-p9jch\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.335744 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.335533    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-dshm\") pod \"canary-v2-kserve-7f65dbbccc-p9jch\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.335744 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.335554    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-p9jch\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.335744 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.335576    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-home\") pod \"canary-v2-kserve-7f65dbbccc-p9jch\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.335744 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.335598    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s9f2\" (UniqueName: \"kubernetes.io/projected/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-kube-api-access-8s9f2\") pod \"canary-v2-kserve-7f65dbbccc-p9jch\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.335973 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.335938    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-p9jch\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.336106 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.336078    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-home\") pod \"canary-v2-kserve-7f65dbbccc-p9jch\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.336219 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.336078    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-p9jch\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.338215 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.338193    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-p9jch\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.338215 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.338197    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-dshm\") pod \"canary-v2-kserve-7f65dbbccc-p9jch\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.346593 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.346572    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s9f2\" (UniqueName: \"kubernetes.io/projected/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-kube-api-access-8s9f2\") pod \"canary-v2-kserve-7f65dbbccc-p9jch\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.378252 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.378223    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:05:50.660627 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.660598    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"]
Sep 16 19:05:50.661684 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:05:50.661661    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf0fb51_77cd_43cc_b72e_e97eb26d9967.slice/crio-f85994c8b34e6b1848f0e98aecba3a20a2bfd6a36277593c0b31b1caedde3100 WatchSource:0}: Error finding container f85994c8b34e6b1848f0e98aecba3a20a2bfd6a36277593c0b31b1caedde3100: Status 404 returned error can't find the container with id f85994c8b34e6b1848f0e98aecba3a20a2bfd6a36277593c0b31b1caedde3100
Sep 16 19:05:50.749685 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.749653    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch" event={"ID":"4cf0fb51-77cd-43cc-b72e-e97eb26d9967","Type":"ContainerStarted","Data":"b5636386c9cd67b7075d687536a9ce5efc347a534d35a9115419c1ee5363c422"}
Sep 16 19:05:50.749804 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.749705    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch" event={"ID":"4cf0fb51-77cd-43cc-b72e-e97eb26d9967","Type":"ContainerStarted","Data":"f85994c8b34e6b1848f0e98aecba3a20a2bfd6a36277593c0b31b1caedde3100"}
Sep 16 19:05:50.770167 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:50.770124    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch" podStartSLOduration=0.770108153 podStartE2EDuration="770.108153ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:50.76907481 +0000 UTC m=+1589.128592804" watchObservedRunningTime="2026-09-16 19:05:50.770108153 +0000 UTC m=+1589.129626147"
Sep 16 19:05:51.378960 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:05:51.378930    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:06:00.378832 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:06:00.378800    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:06:00.391803 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:06:00.391780    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:06:00.822269 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:06:00.822243    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:07:48.431849 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:48.431770    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"]
Sep 16 19:07:49.241111 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:49.241075    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"]
Sep 16 19:07:54.085637 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.085584    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"]
Sep 16 19:07:54.089193 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.089176    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.093026 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.093003    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 16 19:07:54.093908 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.093890    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-n9r7x\""
Sep 16 19:07:54.093985 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.093913    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 16 19:07:54.115472 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.115453    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 16 19:07:54.188527 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.188499    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"]
Sep 16 19:07:54.212596 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.212570    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d39892f-c916-4984-9163-af46c449ddc8-tls-certs\") pod \"test-div-v1-kserve-fb566d667-dkjg8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.212697 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.212600    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-home\") pod \"test-div-v1-kserve-fb566d667-dkjg8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.212697 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.212627    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-dkjg8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.212697 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.212651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77cqs\" (UniqueName: \"kubernetes.io/projected/9d39892f-c916-4984-9163-af46c449ddc8-kube-api-access-77cqs\") pod \"test-div-v1-kserve-fb566d667-dkjg8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.212697 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.212689    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-dshm\") pod \"test-div-v1-kserve-fb566d667-dkjg8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.212861 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.212717    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-model-cache\") pod \"test-div-v1-kserve-fb566d667-dkjg8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.313692 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.313653    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d39892f-c916-4984-9163-af46c449ddc8-tls-certs\") pod \"test-div-v1-kserve-fb566d667-dkjg8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.313933 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.313698    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-home\") pod \"test-div-v1-kserve-fb566d667-dkjg8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.313933 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.313733    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-dkjg8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.313933 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.313816    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77cqs\" (UniqueName: \"kubernetes.io/projected/9d39892f-c916-4984-9163-af46c449ddc8-kube-api-access-77cqs\") pod \"test-div-v1-kserve-fb566d667-dkjg8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.313933 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.313871    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-dshm\") pod \"test-div-v1-kserve-fb566d667-dkjg8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.314288 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.314068    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-model-cache\") pod \"test-div-v1-kserve-fb566d667-dkjg8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.314288 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.314102    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-dkjg8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.314288 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.314174    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-home\") pod \"test-div-v1-kserve-fb566d667-dkjg8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.314512 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.314399    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-model-cache\") pod \"test-div-v1-kserve-fb566d667-dkjg8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.316109 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.316089    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-dshm\") pod \"test-div-v1-kserve-fb566d667-dkjg8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.316544 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.316522    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d39892f-c916-4984-9163-af46c449ddc8-tls-certs\") pod \"test-div-v1-kserve-fb566d667-dkjg8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.345379 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.345312    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77cqs\" (UniqueName: \"kubernetes.io/projected/9d39892f-c916-4984-9163-af46c449ddc8-kube-api-access-77cqs\") pod \"test-div-v1-kserve-fb566d667-dkjg8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.400019 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.399991    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:07:54.702152 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:54.702074    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"]
Sep 16 19:07:54.705751 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:07:54.705719    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d39892f_c916_4984_9163_af46c449ddc8.slice/crio-22192ad204cce86c1d84bc4c53e8dfe182d13d533666290dc283025c1710fb93 WatchSource:0}: Error finding container 22192ad204cce86c1d84bc4c53e8dfe182d13d533666290dc283025c1710fb93: Status 404 returned error can't find the container with id 22192ad204cce86c1d84bc4c53e8dfe182d13d533666290dc283025c1710fb93
Sep 16 19:07:55.436638 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:55.436600    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8" event={"ID":"9d39892f-c916-4984-9163-af46c449ddc8","Type":"ContainerStarted","Data":"edc9d8b1bdb8e40e0333b82d0d066e2f8984da62dda6259214ba12c6625edc8f"}
Sep 16 19:07:55.437065 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:55.436655    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8" event={"ID":"9d39892f-c916-4984-9163-af46c449ddc8","Type":"ContainerStarted","Data":"22192ad204cce86c1d84bc4c53e8dfe182d13d533666290dc283025c1710fb93"}
Sep 16 19:07:55.461495 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:55.461446    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8" podStartSLOduration=1.4614317190000001 podStartE2EDuration="1.461431719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:55.457727591 +0000 UTC m=+1713.817245585" watchObservedRunningTime="2026-09-16 19:07:55.461431719 +0000 UTC m=+1713.820949714"
Sep 16 19:07:56.400218 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:07:56.400181    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:08:03.451461 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.451416    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6" podUID="e9556bd7-a21f-4537-8815-99e31c8112b1" containerName="main" containerID="cri-o://e9535080314043f5a43e6148f6fed474e8b63703847dc0af4a1d61904f85149e" gracePeriod=45
Sep 16 19:08:03.710080 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.710019    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:08:03.893278 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.893246    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9556bd7-a21f-4537-8815-99e31c8112b1-tls-certs\") pod \"e9556bd7-a21f-4537-8815-99e31c8112b1\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") "
Sep 16 19:08:03.893423 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.893292    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvb4m\" (UniqueName: \"kubernetes.io/projected/e9556bd7-a21f-4537-8815-99e31c8112b1-kube-api-access-gvb4m\") pod \"e9556bd7-a21f-4537-8815-99e31c8112b1\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") "
Sep 16 19:08:03.893423 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.893313    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-home\") pod \"e9556bd7-a21f-4537-8815-99e31c8112b1\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") "
Sep 16 19:08:03.893423 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.893332    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-dshm\") pod \"e9556bd7-a21f-4537-8815-99e31c8112b1\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") "
Sep 16 19:08:03.893423 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.893393    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-model-cache\") pod \"e9556bd7-a21f-4537-8815-99e31c8112b1\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") "
Sep 16 19:08:03.893618 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.893457    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-tmp-dir\") pod \"e9556bd7-a21f-4537-8815-99e31c8112b1\" (UID: \"e9556bd7-a21f-4537-8815-99e31c8112b1\") "
Sep 16 19:08:03.893618 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.893542    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-home" (OuterVolumeSpecName: "home") pod "e9556bd7-a21f-4537-8815-99e31c8112b1" (UID: "e9556bd7-a21f-4537-8815-99e31c8112b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:03.893744 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.893727    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:03.893801 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.893743    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9556bd7-a21f-4537-8815-99e31c8112b1" (UID: "e9556bd7-a21f-4537-8815-99e31c8112b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:03.893801 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.893758    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9556bd7-a21f-4537-8815-99e31c8112b1" (UID: "e9556bd7-a21f-4537-8815-99e31c8112b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:03.895482 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.895452    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-dshm" (OuterVolumeSpecName: "dshm") pod "e9556bd7-a21f-4537-8815-99e31c8112b1" (UID: "e9556bd7-a21f-4537-8815-99e31c8112b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:03.895586 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.895504    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9556bd7-a21f-4537-8815-99e31c8112b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9556bd7-a21f-4537-8815-99e31c8112b1" (UID: "e9556bd7-a21f-4537-8815-99e31c8112b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:03.895586 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.895529    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9556bd7-a21f-4537-8815-99e31c8112b1-kube-api-access-gvb4m" (OuterVolumeSpecName: "kube-api-access-gvb4m") pod "e9556bd7-a21f-4537-8815-99e31c8112b1" (UID: "e9556bd7-a21f-4537-8815-99e31c8112b1"). InnerVolumeSpecName "kube-api-access-gvb4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:03.994205 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.994180    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:03.994205 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.994204    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9556bd7-a21f-4537-8815-99e31c8112b1-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:03.994338 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.994213    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvb4m\" (UniqueName: \"kubernetes.io/projected/e9556bd7-a21f-4537-8815-99e31c8112b1-kube-api-access-gvb4m\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:03.994338 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.994221    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:03.994338 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:03.994230    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9556bd7-a21f-4537-8815-99e31c8112b1-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:04.259809 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.259744    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch" podUID="4cf0fb51-77cd-43cc-b72e-e97eb26d9967" containerName="main" containerID="cri-o://b5636386c9cd67b7075d687536a9ce5efc347a534d35a9115419c1ee5363c422" gracePeriod=45
Sep 16 19:08:04.400584 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.400547    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:08:04.414149 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.414124    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:08:04.487657 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.487625    2565 generic.go:356] "Generic (PLEG): container finished" podID="4cf0fb51-77cd-43cc-b72e-e97eb26d9967" containerID="b5636386c9cd67b7075d687536a9ce5efc347a534d35a9115419c1ee5363c422" exitCode=0
Sep 16 19:08:04.488032 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.487696    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch" event={"ID":"4cf0fb51-77cd-43cc-b72e-e97eb26d9967","Type":"ContainerDied","Data":"b5636386c9cd67b7075d687536a9ce5efc347a534d35a9115419c1ee5363c422"}
Sep 16 19:08:04.489209 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.489187    2565 generic.go:356] "Generic (PLEG): container finished" podID="e9556bd7-a21f-4537-8815-99e31c8112b1" containerID="e9535080314043f5a43e6148f6fed474e8b63703847dc0af4a1d61904f85149e" exitCode=0
Sep 16 19:08:04.489357 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.489231    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"
Sep 16 19:08:04.489357 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.489269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6" event={"ID":"e9556bd7-a21f-4537-8815-99e31c8112b1","Type":"ContainerDied","Data":"e9535080314043f5a43e6148f6fed474e8b63703847dc0af4a1d61904f85149e"}
Sep 16 19:08:04.489357 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.489320    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6" event={"ID":"e9556bd7-a21f-4537-8815-99e31c8112b1","Type":"ContainerDied","Data":"6cf92a064eb535a8139ed07aa4b58d416c9de388d4da057a434e9ecb3eea8722"}
Sep 16 19:08:04.489536 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.489364    2565 scope.go:117] "RemoveContainer" containerID="e9535080314043f5a43e6148f6fed474e8b63703847dc0af4a1d61904f85149e"
Sep 16 19:08:04.500098 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.500077    2565 scope.go:117] "RemoveContainer" containerID="e9535080314043f5a43e6148f6fed474e8b63703847dc0af4a1d61904f85149e"
Sep 16 19:08:04.500424 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:08:04.500406    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9535080314043f5a43e6148f6fed474e8b63703847dc0af4a1d61904f85149e\": container with ID starting with e9535080314043f5a43e6148f6fed474e8b63703847dc0af4a1d61904f85149e not found: ID does not exist" containerID="e9535080314043f5a43e6148f6fed474e8b63703847dc0af4a1d61904f85149e"
Sep 16 19:08:04.500491 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.500433    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9535080314043f5a43e6148f6fed474e8b63703847dc0af4a1d61904f85149e"} err="failed to get container status \"e9535080314043f5a43e6148f6fed474e8b63703847dc0af4a1d61904f85149e\": rpc error: code = NotFound desc = could not find container \"e9535080314043f5a43e6148f6fed474e8b63703847dc0af4a1d61904f85149e\": container with ID starting with e9535080314043f5a43e6148f6fed474e8b63703847dc0af4a1d61904f85149e not found: ID does not exist"
Sep 16 19:08:04.504185 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.504165    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:08:04.515296 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.515241    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"]
Sep 16 19:08:04.518849 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.518829    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-hctx6"]
Sep 16 19:08:04.525082 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.525066    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:08:04.699936 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.699905    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-model-cache\") pod \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") "
Sep 16 19:08:04.700107 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.699951    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-home\") pod \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") "
Sep 16 19:08:04.700107 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.699969    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-tls-certs\") pod \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") "
Sep 16 19:08:04.700107 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.699988    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-tmp-dir\") pod \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") "
Sep 16 19:08:04.700107 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.700018    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s9f2\" (UniqueName: \"kubernetes.io/projected/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-kube-api-access-8s9f2\") pod \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") "
Sep 16 19:08:04.700107 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.700066    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-dshm\") pod \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\" (UID: \"4cf0fb51-77cd-43cc-b72e-e97eb26d9967\") "
Sep 16 19:08:04.700370 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.700215    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-home" (OuterVolumeSpecName: "home") pod "4cf0fb51-77cd-43cc-b72e-e97eb26d9967" (UID: "4cf0fb51-77cd-43cc-b72e-e97eb26d9967"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:04.700370 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.700225    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-model-cache" (OuterVolumeSpecName: "model-cache") pod "4cf0fb51-77cd-43cc-b72e-e97eb26d9967" (UID: "4cf0fb51-77cd-43cc-b72e-e97eb26d9967"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:04.700370 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.700247    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4cf0fb51-77cd-43cc-b72e-e97eb26d9967" (UID: "4cf0fb51-77cd-43cc-b72e-e97eb26d9967"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:04.700370 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.700367    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:04.700508 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.700382    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:04.700508 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.700391    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:04.702138 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.702115    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-dshm" (OuterVolumeSpecName: "dshm") pod "4cf0fb51-77cd-43cc-b72e-e97eb26d9967" (UID: "4cf0fb51-77cd-43cc-b72e-e97eb26d9967"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:04.702211 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.702195    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4cf0fb51-77cd-43cc-b72e-e97eb26d9967" (UID: "4cf0fb51-77cd-43cc-b72e-e97eb26d9967"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:04.702260 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.702199    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-kube-api-access-8s9f2" (OuterVolumeSpecName: "kube-api-access-8s9f2") pod "4cf0fb51-77cd-43cc-b72e-e97eb26d9967" (UID: "4cf0fb51-77cd-43cc-b72e-e97eb26d9967"). InnerVolumeSpecName "kube-api-access-8s9f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:04.801718 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.801656    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:04.801718 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.801681    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s9f2\" (UniqueName: \"kubernetes.io/projected/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-kube-api-access-8s9f2\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:04.801718 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.801696    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf0fb51-77cd-43cc-b72e-e97eb26d9967-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:04.866069 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.866042    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"]
Sep 16 19:08:04.866439 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.866426    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9556bd7-a21f-4537-8815-99e31c8112b1" containerName="main"
Sep 16 19:08:04.866493 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.866441    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9556bd7-a21f-4537-8815-99e31c8112b1" containerName="main"
Sep 16 19:08:04.866493 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.866456    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf0fb51-77cd-43cc-b72e-e97eb26d9967" containerName="main"
Sep 16 19:08:04.866493 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.866462    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf0fb51-77cd-43cc-b72e-e97eb26d9967" containerName="main"
Sep 16 19:08:04.866600 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.866534    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf0fb51-77cd-43cc-b72e-e97eb26d9967" containerName="main"
Sep 16 19:08:04.866600 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.866541    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9556bd7-a21f-4537-8815-99e31c8112b1" containerName="main"
Sep 16 19:08:04.869577 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.869562    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:04.873064 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.873047    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 16 19:08:04.888648 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:04.888623    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"]
Sep 16 19:08:05.003730 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.003702    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8babfd6e-72eb-4383-a20c-67b95af23058-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-qmg42\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:05.003872 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.003744    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-qmg42\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:05.003872 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.003762    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-qmg42\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:05.003872 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.003834    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nddrw\" (UniqueName: \"kubernetes.io/projected/8babfd6e-72eb-4383-a20c-67b95af23058-kube-api-access-nddrw\") pod \"test-div-v2-kserve-5f6fdb6b77-qmg42\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:05.004046 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.003920    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-home\") pod \"test-div-v2-kserve-5f6fdb6b77-qmg42\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:05.004046 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.003969    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-qmg42\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:05.105157 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.105070    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-home\") pod \"test-div-v2-kserve-5f6fdb6b77-qmg42\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:05.105157 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.105131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-qmg42\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:05.105330 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.105179    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8babfd6e-72eb-4383-a20c-67b95af23058-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-qmg42\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:05.105330 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.105214    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-qmg42\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:05.105330 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.105239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-qmg42\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:05.105330 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.105300    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nddrw\" (UniqueName: \"kubernetes.io/projected/8babfd6e-72eb-4383-a20c-67b95af23058-kube-api-access-nddrw\") pod \"test-div-v2-kserve-5f6fdb6b77-qmg42\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:05.105574 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.105543    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-home\") pod \"test-div-v2-kserve-5f6fdb6b77-qmg42\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:05.105574 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.105565    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-qmg42\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:05.105744 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.105628    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-qmg42\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:05.107486 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.107461    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-qmg42\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:05.107672 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.107655    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8babfd6e-72eb-4383-a20c-67b95af23058-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-qmg42\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:05.116703 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.116656    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nddrw\" (UniqueName: \"kubernetes.io/projected/8babfd6e-72eb-4383-a20c-67b95af23058-kube-api-access-nddrw\") pod \"test-div-v2-kserve-5f6fdb6b77-qmg42\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:05.179254 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.179229    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:05.460409 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.460383    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"]
Sep 16 19:08:05.462113 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:08:05.462084    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8babfd6e_72eb_4383_a20c_67b95af23058.slice/crio-1d04ad7016cdd6d3a7e7732dd634fbfd35e6bc1acd6b0710fa17ed8dd8b2e804 WatchSource:0}: Error finding container 1d04ad7016cdd6d3a7e7732dd634fbfd35e6bc1acd6b0710fa17ed8dd8b2e804: Status 404 returned error can't find the container with id 1d04ad7016cdd6d3a7e7732dd634fbfd35e6bc1acd6b0710fa17ed8dd8b2e804
Sep 16 19:08:05.495657 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.495635    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"
Sep 16 19:08:05.495657 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.495644    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch" event={"ID":"4cf0fb51-77cd-43cc-b72e-e97eb26d9967","Type":"ContainerDied","Data":"f85994c8b34e6b1848f0e98aecba3a20a2bfd6a36277593c0b31b1caedde3100"}
Sep 16 19:08:05.496105 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.495702    2565 scope.go:117] "RemoveContainer" containerID="b5636386c9cd67b7075d687536a9ce5efc347a534d35a9115419c1ee5363c422"
Sep 16 19:08:05.497407 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.497376    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42" event={"ID":"8babfd6e-72eb-4383-a20c-67b95af23058","Type":"ContainerStarted","Data":"1d04ad7016cdd6d3a7e7732dd634fbfd35e6bc1acd6b0710fa17ed8dd8b2e804"}
Sep 16 19:08:05.524503 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.524480    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"]
Sep 16 19:08:05.533940 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:05.533919    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-p9jch"]
Sep 16 19:08:06.207239 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:06.207205    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cf0fb51-77cd-43cc-b72e-e97eb26d9967" path="/var/lib/kubelet/pods/4cf0fb51-77cd-43cc-b72e-e97eb26d9967/volumes"
Sep 16 19:08:06.207636 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:06.207621    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9556bd7-a21f-4537-8815-99e31c8112b1" path="/var/lib/kubelet/pods/e9556bd7-a21f-4537-8815-99e31c8112b1/volumes"
Sep 16 19:08:06.508194 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:06.508159    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42" event={"ID":"8babfd6e-72eb-4383-a20c-67b95af23058","Type":"ContainerStarted","Data":"6e9584208ccf28491080c620b3fd3acbf32f71acd230991c71dbee6a2037241a"}
Sep 16 19:08:06.548194 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:06.548147    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42" podStartSLOduration=2.548131615 podStartE2EDuration="2.548131615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:06.547092124 +0000 UTC m=+1724.906610120" watchObservedRunningTime="2026-09-16 19:08:06.548131615 +0000 UTC m=+1724.907649610"
Sep 16 19:08:07.180032 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:07.179987    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:15.180202 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:15.180168    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:15.192541 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:15.192513    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:15.567428 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:15.567401    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:29.153909 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:29.153871    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"]
Sep 16 19:08:29.917828 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:29.917795    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"]
Sep 16 19:08:40.588714 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.588677    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"]
Sep 16 19:08:40.592585 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.592562    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:40.598472 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.598437    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-kkmjc\""
Sep 16 19:08:40.598585 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.598478    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 16 19:08:40.598649 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.598584    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 16 19:08:40.598707 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.598652    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 16 19:08:40.635070 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.635044    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"]
Sep 16 19:08:40.669459 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.669433    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkdxw\" (UniqueName: \"kubernetes.io/projected/a51c130b-11d9-4b77-99ea-3d152d1db2ea-kube-api-access-wkdxw\") pod \"test-div-v1-kserve-fb566d667-nx25c\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:40.669564 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.669464    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-home\") pod \"test-div-v1-kserve-fb566d667-nx25c\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:40.669564 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.669480    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-nx25c\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:40.669564 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.669521    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-dshm\") pod \"test-div-v1-kserve-fb566d667-nx25c\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:40.669564 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.669556    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-model-cache\") pod \"test-div-v1-kserve-fb566d667-nx25c\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:40.669738 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.669585    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a51c130b-11d9-4b77-99ea-3d152d1db2ea-tls-certs\") pod \"test-div-v1-kserve-fb566d667-nx25c\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:40.770759 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.770727    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-dshm\") pod \"test-div-v1-kserve-fb566d667-nx25c\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:40.770925 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.770781    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-model-cache\") pod \"test-div-v1-kserve-fb566d667-nx25c\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:40.770925 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.770807    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a51c130b-11d9-4b77-99ea-3d152d1db2ea-tls-certs\") pod \"test-div-v1-kserve-fb566d667-nx25c\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:40.770925 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.770847    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdxw\" (UniqueName: \"kubernetes.io/projected/a51c130b-11d9-4b77-99ea-3d152d1db2ea-kube-api-access-wkdxw\") pod \"test-div-v1-kserve-fb566d667-nx25c\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:40.771107 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.770968    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-home\") pod \"test-div-v1-kserve-fb566d667-nx25c\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:40.771107 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.771004    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-nx25c\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:40.771251 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.771229    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-model-cache\") pod \"test-div-v1-kserve-fb566d667-nx25c\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:40.771409 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.771385    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-home\") pod \"test-div-v1-kserve-fb566d667-nx25c\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:40.771474 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.771399    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-nx25c\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:40.773024 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.773006    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-dshm\") pod \"test-div-v1-kserve-fb566d667-nx25c\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:40.773356 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.773328    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a51c130b-11d9-4b77-99ea-3d152d1db2ea-tls-certs\") pod \"test-div-v1-kserve-fb566d667-nx25c\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:40.786080 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.786051    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkdxw\" (UniqueName: \"kubernetes.io/projected/a51c130b-11d9-4b77-99ea-3d152d1db2ea-kube-api-access-wkdxw\") pod \"test-div-v1-kserve-fb566d667-nx25c\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:40.902310 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:40.902250    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:41.229687 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:41.229637    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"]
Sep 16 19:08:41.232769 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:08:41.232734    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51c130b_11d9_4b77_99ea_3d152d1db2ea.slice/crio-24f9e41f0f74234483f39aaa980d0f5738f7414b310599c7803b0af69db237ee WatchSource:0}: Error finding container 24f9e41f0f74234483f39aaa980d0f5738f7414b310599c7803b0af69db237ee: Status 404 returned error can't find the container with id 24f9e41f0f74234483f39aaa980d0f5738f7414b310599c7803b0af69db237ee
Sep 16 19:08:41.699194 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:41.699109    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c" event={"ID":"a51c130b-11d9-4b77-99ea-3d152d1db2ea","Type":"ContainerStarted","Data":"dcfa080affce6d31dfdca2ad2214034b6b5fbc4947ab4653ee98e22cb85c0d81"}
Sep 16 19:08:41.699194 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:41.699168    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c" event={"ID":"a51c130b-11d9-4b77-99ea-3d152d1db2ea","Type":"ContainerStarted","Data":"24f9e41f0f74234483f39aaa980d0f5738f7414b310599c7803b0af69db237ee"}
Sep 16 19:08:41.742415 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:41.742331    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c" podStartSLOduration=1.742310461 podStartE2EDuration="1.742310461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:41.740899404 +0000 UTC m=+1760.100417400" watchObservedRunningTime="2026-09-16 19:08:41.742310461 +0000 UTC m=+1760.101828459"
Sep 16 19:08:41.902779 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:41.902746    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:44.172723 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.172690    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8" podUID="9d39892f-c916-4984-9163-af46c449ddc8" containerName="main" containerID="cri-o://edc9d8b1bdb8e40e0333b82d0d066e2f8984da62dda6259214ba12c6625edc8f" gracePeriod=45
Sep 16 19:08:44.432352 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.432289    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:08:44.500907 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.500872    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-home\") pod \"9d39892f-c916-4984-9163-af46c449ddc8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") "
Sep 16 19:08:44.501083 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.500934    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-dshm\") pod \"9d39892f-c916-4984-9163-af46c449ddc8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") "
Sep 16 19:08:44.501083 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.500968    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-tmp-dir\") pod \"9d39892f-c916-4984-9163-af46c449ddc8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") "
Sep 16 19:08:44.501083 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.501023    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-model-cache\") pod \"9d39892f-c916-4984-9163-af46c449ddc8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") "
Sep 16 19:08:44.501083 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.501066    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77cqs\" (UniqueName: \"kubernetes.io/projected/9d39892f-c916-4984-9163-af46c449ddc8-kube-api-access-77cqs\") pod \"9d39892f-c916-4984-9163-af46c449ddc8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") "
Sep 16 19:08:44.501287 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.501113    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d39892f-c916-4984-9163-af46c449ddc8-tls-certs\") pod \"9d39892f-c916-4984-9163-af46c449ddc8\" (UID: \"9d39892f-c916-4984-9163-af46c449ddc8\") "
Sep 16 19:08:44.501287 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.501146    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-home" (OuterVolumeSpecName: "home") pod "9d39892f-c916-4984-9163-af46c449ddc8" (UID: "9d39892f-c916-4984-9163-af46c449ddc8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:44.501399 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.501271    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d39892f-c916-4984-9163-af46c449ddc8" (UID: "9d39892f-c916-4984-9163-af46c449ddc8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:44.501399 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.501321    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d39892f-c916-4984-9163-af46c449ddc8" (UID: "9d39892f-c916-4984-9163-af46c449ddc8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:44.501478 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.501432    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:44.501478 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.501450    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:44.501478 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.501465    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:44.503165 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.503138    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-dshm" (OuterVolumeSpecName: "dshm") pod "9d39892f-c916-4984-9163-af46c449ddc8" (UID: "9d39892f-c916-4984-9163-af46c449ddc8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:44.503279 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.503182    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d39892f-c916-4984-9163-af46c449ddc8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d39892f-c916-4984-9163-af46c449ddc8" (UID: "9d39892f-c916-4984-9163-af46c449ddc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:44.503279 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.503214    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d39892f-c916-4984-9163-af46c449ddc8-kube-api-access-77cqs" (OuterVolumeSpecName: "kube-api-access-77cqs") pod "9d39892f-c916-4984-9163-af46c449ddc8" (UID: "9d39892f-c916-4984-9163-af46c449ddc8"). InnerVolumeSpecName "kube-api-access-77cqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:44.602238 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.602211    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77cqs\" (UniqueName: \"kubernetes.io/projected/9d39892f-c916-4984-9163-af46c449ddc8-kube-api-access-77cqs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:44.602238 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.602238    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d39892f-c916-4984-9163-af46c449ddc8-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:44.602398 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.602251    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d39892f-c916-4984-9163-af46c449ddc8-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:44.716849 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.716791    2565 generic.go:356] "Generic (PLEG): container finished" podID="9d39892f-c916-4984-9163-af46c449ddc8" containerID="edc9d8b1bdb8e40e0333b82d0d066e2f8984da62dda6259214ba12c6625edc8f" exitCode=0
Sep 16 19:08:44.716944 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.716849    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"
Sep 16 19:08:44.716944 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.716861    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8" event={"ID":"9d39892f-c916-4984-9163-af46c449ddc8","Type":"ContainerDied","Data":"edc9d8b1bdb8e40e0333b82d0d066e2f8984da62dda6259214ba12c6625edc8f"}
Sep 16 19:08:44.716944 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.716926    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8" event={"ID":"9d39892f-c916-4984-9163-af46c449ddc8","Type":"ContainerDied","Data":"22192ad204cce86c1d84bc4c53e8dfe182d13d533666290dc283025c1710fb93"}
Sep 16 19:08:44.717061 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.716947    2565 scope.go:117] "RemoveContainer" containerID="edc9d8b1bdb8e40e0333b82d0d066e2f8984da62dda6259214ba12c6625edc8f"
Sep 16 19:08:44.726048 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.726033    2565 scope.go:117] "RemoveContainer" containerID="edc9d8b1bdb8e40e0333b82d0d066e2f8984da62dda6259214ba12c6625edc8f"
Sep 16 19:08:44.726297 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:08:44.726280    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc9d8b1bdb8e40e0333b82d0d066e2f8984da62dda6259214ba12c6625edc8f\": container with ID starting with edc9d8b1bdb8e40e0333b82d0d066e2f8984da62dda6259214ba12c6625edc8f not found: ID does not exist" containerID="edc9d8b1bdb8e40e0333b82d0d066e2f8984da62dda6259214ba12c6625edc8f"
Sep 16 19:08:44.726369 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.726305    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc9d8b1bdb8e40e0333b82d0d066e2f8984da62dda6259214ba12c6625edc8f"} err="failed to get container status \"edc9d8b1bdb8e40e0333b82d0d066e2f8984da62dda6259214ba12c6625edc8f\": rpc error: code = NotFound desc = could not find container \"edc9d8b1bdb8e40e0333b82d0d066e2f8984da62dda6259214ba12c6625edc8f\": container with ID starting with edc9d8b1bdb8e40e0333b82d0d066e2f8984da62dda6259214ba12c6625edc8f not found: ID does not exist"
Sep 16 19:08:44.760644 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.760621    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"]
Sep 16 19:08:44.764593 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.764573    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-dkjg8"]
Sep 16 19:08:44.936290 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:44.936264    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42" podUID="8babfd6e-72eb-4383-a20c-67b95af23058" containerName="main" containerID="cri-o://6e9584208ccf28491080c620b3fd3acbf32f71acd230991c71dbee6a2037241a" gracePeriod=45
Sep 16 19:08:45.181158 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.181132    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:45.309365 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.309276    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-tmp-dir\") pod \"8babfd6e-72eb-4383-a20c-67b95af23058\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") "
Sep 16 19:08:45.309365 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.309308    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-home\") pod \"8babfd6e-72eb-4383-a20c-67b95af23058\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") "
Sep 16 19:08:45.309365 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.309334    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8babfd6e-72eb-4383-a20c-67b95af23058-tls-certs\") pod \"8babfd6e-72eb-4383-a20c-67b95af23058\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") "
Sep 16 19:08:45.309633 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.309410    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-dshm\") pod \"8babfd6e-72eb-4383-a20c-67b95af23058\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") "
Sep 16 19:08:45.309633 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.309425    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-model-cache\") pod \"8babfd6e-72eb-4383-a20c-67b95af23058\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") "
Sep 16 19:08:45.309633 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.309443    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nddrw\" (UniqueName: \"kubernetes.io/projected/8babfd6e-72eb-4383-a20c-67b95af23058-kube-api-access-nddrw\") pod \"8babfd6e-72eb-4383-a20c-67b95af23058\" (UID: \"8babfd6e-72eb-4383-a20c-67b95af23058\") "
Sep 16 19:08:45.309633 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.309590    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8babfd6e-72eb-4383-a20c-67b95af23058" (UID: "8babfd6e-72eb-4383-a20c-67b95af23058"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:45.309633 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.309603    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-home" (OuterVolumeSpecName: "home") pod "8babfd6e-72eb-4383-a20c-67b95af23058" (UID: "8babfd6e-72eb-4383-a20c-67b95af23058"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:45.309902 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.309673    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-model-cache" (OuterVolumeSpecName: "model-cache") pod "8babfd6e-72eb-4383-a20c-67b95af23058" (UID: "8babfd6e-72eb-4383-a20c-67b95af23058"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:45.309902 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.309750    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:45.309902 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.309769    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:45.309902 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.309782    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:45.311491 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.311468    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8babfd6e-72eb-4383-a20c-67b95af23058-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8babfd6e-72eb-4383-a20c-67b95af23058" (UID: "8babfd6e-72eb-4383-a20c-67b95af23058"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:45.311619 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.311605    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-dshm" (OuterVolumeSpecName: "dshm") pod "8babfd6e-72eb-4383-a20c-67b95af23058" (UID: "8babfd6e-72eb-4383-a20c-67b95af23058"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:45.311685 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.311604    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8babfd6e-72eb-4383-a20c-67b95af23058-kube-api-access-nddrw" (OuterVolumeSpecName: "kube-api-access-nddrw") pod "8babfd6e-72eb-4383-a20c-67b95af23058" (UID: "8babfd6e-72eb-4383-a20c-67b95af23058"). InnerVolumeSpecName "kube-api-access-nddrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:45.410709 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.410677    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8babfd6e-72eb-4383-a20c-67b95af23058-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:45.410709 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.410704    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8babfd6e-72eb-4383-a20c-67b95af23058-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:45.410709 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.410714    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nddrw\" (UniqueName: \"kubernetes.io/projected/8babfd6e-72eb-4383-a20c-67b95af23058-kube-api-access-nddrw\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:08:45.723959 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.723919    2565 generic.go:356] "Generic (PLEG): container finished" podID="8babfd6e-72eb-4383-a20c-67b95af23058" containerID="6e9584208ccf28491080c620b3fd3acbf32f71acd230991c71dbee6a2037241a" exitCode=0
Sep 16 19:08:45.724141 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.723969    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"
Sep 16 19:08:45.724141 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.724004    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42" event={"ID":"8babfd6e-72eb-4383-a20c-67b95af23058","Type":"ContainerDied","Data":"6e9584208ccf28491080c620b3fd3acbf32f71acd230991c71dbee6a2037241a"}
Sep 16 19:08:45.724141 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.724055    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42" event={"ID":"8babfd6e-72eb-4383-a20c-67b95af23058","Type":"ContainerDied","Data":"1d04ad7016cdd6d3a7e7732dd634fbfd35e6bc1acd6b0710fa17ed8dd8b2e804"}
Sep 16 19:08:45.724141 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.724083    2565 scope.go:117] "RemoveContainer" containerID="6e9584208ccf28491080c620b3fd3acbf32f71acd230991c71dbee6a2037241a"
Sep 16 19:08:45.733622 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.733603    2565 scope.go:117] "RemoveContainer" containerID="6e9584208ccf28491080c620b3fd3acbf32f71acd230991c71dbee6a2037241a"
Sep 16 19:08:45.733905 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:08:45.733886    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e9584208ccf28491080c620b3fd3acbf32f71acd230991c71dbee6a2037241a\": container with ID starting with 6e9584208ccf28491080c620b3fd3acbf32f71acd230991c71dbee6a2037241a not found: ID does not exist" containerID="6e9584208ccf28491080c620b3fd3acbf32f71acd230991c71dbee6a2037241a"
Sep 16 19:08:45.733959 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.733914    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e9584208ccf28491080c620b3fd3acbf32f71acd230991c71dbee6a2037241a"} err="failed to get container status \"6e9584208ccf28491080c620b3fd3acbf32f71acd230991c71dbee6a2037241a\": rpc error: code = NotFound desc = could not find container \"6e9584208ccf28491080c620b3fd3acbf32f71acd230991c71dbee6a2037241a\": container with ID starting with 6e9584208ccf28491080c620b3fd3acbf32f71acd230991c71dbee6a2037241a not found: ID does not exist"
Sep 16 19:08:45.747334 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.747310    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"]
Sep 16 19:08:45.753130 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:45.753110    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-qmg42"]
Sep 16 19:08:46.207583 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:46.207518    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8babfd6e-72eb-4383-a20c-67b95af23058" path="/var/lib/kubelet/pods/8babfd6e-72eb-4383-a20c-67b95af23058/volumes"
Sep 16 19:08:46.207900 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:46.207887    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d39892f-c916-4984-9163-af46c449ddc8" path="/var/lib/kubelet/pods/9d39892f-c916-4984-9163-af46c449ddc8/volumes"
Sep 16 19:08:50.903488 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:50.903450    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:50.916127 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:50.916104    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:08:51.769657 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:08:51.769626    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:09:00.667415 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.667380    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"]
Sep 16 19:09:00.667969 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.667948    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8babfd6e-72eb-4383-a20c-67b95af23058" containerName="main"
Sep 16 19:09:00.668048 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.667972    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8babfd6e-72eb-4383-a20c-67b95af23058" containerName="main"
Sep 16 19:09:00.668048 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.667997    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d39892f-c916-4984-9163-af46c449ddc8" containerName="main"
Sep 16 19:09:00.668048 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.668005    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d39892f-c916-4984-9163-af46c449ddc8" containerName="main"
Sep 16 19:09:00.668199 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.668114    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d39892f-c916-4984-9163-af46c449ddc8" containerName="main"
Sep 16 19:09:00.668199 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.668134    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8babfd6e-72eb-4383-a20c-67b95af23058" containerName="main"
Sep 16 19:09:00.670920 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.670899    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:00.673473 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.673456    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 16 19:09:00.685861 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.685827    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"]
Sep 16 19:09:00.727695 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.727662    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-home\") pod \"test-div-v2-kserve-5f6fdb6b77-jr8cr\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:00.727825 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.727710    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-jr8cr\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:00.727825 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.727808    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9fss\" (UniqueName: \"kubernetes.io/projected/34684c6d-ecca-42fe-a6df-052c74193c88-kube-api-access-k9fss\") pod \"test-div-v2-kserve-5f6fdb6b77-jr8cr\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:00.727945 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.727864    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-jr8cr\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:00.728006 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.727971    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34684c6d-ecca-42fe-a6df-052c74193c88-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-jr8cr\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:00.728068 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.728010    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-jr8cr\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:00.828598 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.828568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9fss\" (UniqueName: \"kubernetes.io/projected/34684c6d-ecca-42fe-a6df-052c74193c88-kube-api-access-k9fss\") pod \"test-div-v2-kserve-5f6fdb6b77-jr8cr\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:00.828735 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.828615    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-jr8cr\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:00.828793 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.828728    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34684c6d-ecca-42fe-a6df-052c74193c88-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-jr8cr\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:00.828793 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.828772    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-jr8cr\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:00.828889 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.828811    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-home\") pod \"test-div-v2-kserve-5f6fdb6b77-jr8cr\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:00.828889 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.828842    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-jr8cr\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:00.828992 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.828969    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-jr8cr\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:00.829142 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.829107    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-jr8cr\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:00.829295 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.829274    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-home\") pod \"test-div-v2-kserve-5f6fdb6b77-jr8cr\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:00.830873 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.830852    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-jr8cr\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:00.831154 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.831139    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34684c6d-ecca-42fe-a6df-052c74193c88-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-jr8cr\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:00.840838 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.840815    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9fss\" (UniqueName: \"kubernetes.io/projected/34684c6d-ecca-42fe-a6df-052c74193c88-kube-api-access-k9fss\") pod \"test-div-v2-kserve-5f6fdb6b77-jr8cr\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:00.981256 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:00.981228    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:01.268668 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:01.268644    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"]
Sep 16 19:09:01.270445 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:09:01.270419    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34684c6d_ecca_42fe_a6df_052c74193c88.slice/crio-2953fadaf7f058d47849819aeebf2a4fa0ea0d978a3cf2e91697b536cf6f1a8e WatchSource:0}: Error finding container 2953fadaf7f058d47849819aeebf2a4fa0ea0d978a3cf2e91697b536cf6f1a8e: Status 404 returned error can't find the container with id 2953fadaf7f058d47849819aeebf2a4fa0ea0d978a3cf2e91697b536cf6f1a8e
Sep 16 19:09:01.815737 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:01.815695    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr" event={"ID":"34684c6d-ecca-42fe-a6df-052c74193c88","Type":"ContainerStarted","Data":"7ff6fb1c5c238a7dd6763468ec59078291544191c81d1f2d8848b3b428d3dca0"}
Sep 16 19:09:01.816119 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:01.815798    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr" event={"ID":"34684c6d-ecca-42fe-a6df-052c74193c88","Type":"ContainerStarted","Data":"2953fadaf7f058d47849819aeebf2a4fa0ea0d978a3cf2e91697b536cf6f1a8e"}
Sep 16 19:09:01.841127 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:01.841070    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr" podStartSLOduration=1.841054866 podStartE2EDuration="1.841054866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:01.838431863 +0000 UTC m=+1780.197949857" watchObservedRunningTime="2026-09-16 19:09:01.841054866 +0000 UTC m=+1780.200572895"
Sep 16 19:09:01.981920 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:01.981879    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:10.982314 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:10.982282    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:10.994419 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:10.994398    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:11.879861 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:11.879831    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:22.721935 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:22.721858    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"]
Sep 16 19:09:22.882243 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:22.882205    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"]
Sep 16 19:09:37.741130 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:37.741068    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr" podUID="34684c6d-ecca-42fe-a6df-052c74193c88" containerName="main" containerID="cri-o://7ff6fb1c5c238a7dd6763468ec59078291544191c81d1f2d8848b3b428d3dca0" gracePeriod=45
Sep 16 19:09:37.869356 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:37.869304    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr" podUID="34684c6d-ecca-42fe-a6df-052c74193c88" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 16 19:09:37.901464 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:37.901408    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c" podUID="a51c130b-11d9-4b77-99ea-3d152d1db2ea" containerName="main" containerID="cri-o://dcfa080affce6d31dfdca2ad2214034b6b5fbc4947ab4653ee98e22cb85c0d81" gracePeriod=45
Sep 16 19:09:38.009721 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.009692    2565 generic.go:356] "Generic (PLEG): container finished" podID="34684c6d-ecca-42fe-a6df-052c74193c88" containerID="7ff6fb1c5c238a7dd6763468ec59078291544191c81d1f2d8848b3b428d3dca0" exitCode=0
Sep 16 19:09:38.009842 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.009771    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr" event={"ID":"34684c6d-ecca-42fe-a6df-052c74193c88","Type":"ContainerDied","Data":"7ff6fb1c5c238a7dd6763468ec59078291544191c81d1f2d8848b3b428d3dca0"}
Sep 16 19:09:38.026893 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.026870    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:38.126930 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.126895    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-home\") pod \"34684c6d-ecca-42fe-a6df-052c74193c88\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") "
Sep 16 19:09:38.127067 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.126954    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-model-cache\") pod \"34684c6d-ecca-42fe-a6df-052c74193c88\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") "
Sep 16 19:09:38.127067 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.126992    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-dshm\") pod \"34684c6d-ecca-42fe-a6df-052c74193c88\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") "
Sep 16 19:09:38.127067 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.127026    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9fss\" (UniqueName: \"kubernetes.io/projected/34684c6d-ecca-42fe-a6df-052c74193c88-kube-api-access-k9fss\") pod \"34684c6d-ecca-42fe-a6df-052c74193c88\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") "
Sep 16 19:09:38.127249 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.127100    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-tmp-dir\") pod \"34684c6d-ecca-42fe-a6df-052c74193c88\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") "
Sep 16 19:09:38.127249 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.127122    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-home" (OuterVolumeSpecName: "home") pod "34684c6d-ecca-42fe-a6df-052c74193c88" (UID: "34684c6d-ecca-42fe-a6df-052c74193c88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:38.127249 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.127146    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34684c6d-ecca-42fe-a6df-052c74193c88-tls-certs\") pod \"34684c6d-ecca-42fe-a6df-052c74193c88\" (UID: \"34684c6d-ecca-42fe-a6df-052c74193c88\") "
Sep 16 19:09:38.127439 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.127260    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-model-cache" (OuterVolumeSpecName: "model-cache") pod "34684c6d-ecca-42fe-a6df-052c74193c88" (UID: "34684c6d-ecca-42fe-a6df-052c74193c88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:38.127439 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.127404    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34684c6d-ecca-42fe-a6df-052c74193c88" (UID: "34684c6d-ecca-42fe-a6df-052c74193c88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:38.127558 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.127533    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:09:38.127619 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.127560    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:09:38.127619 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.127575    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:09:38.129207 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.129172    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34684c6d-ecca-42fe-a6df-052c74193c88-kube-api-access-k9fss" (OuterVolumeSpecName: "kube-api-access-k9fss") pod "34684c6d-ecca-42fe-a6df-052c74193c88" (UID: "34684c6d-ecca-42fe-a6df-052c74193c88"). InnerVolumeSpecName "kube-api-access-k9fss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:38.129207 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.129178    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-dshm" (OuterVolumeSpecName: "dshm") pod "34684c6d-ecca-42fe-a6df-052c74193c88" (UID: "34684c6d-ecca-42fe-a6df-052c74193c88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:38.129366 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.129218    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34684c6d-ecca-42fe-a6df-052c74193c88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34684c6d-ecca-42fe-a6df-052c74193c88" (UID: "34684c6d-ecca-42fe-a6df-052c74193c88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:38.143169 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.143151    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:09:38.228354 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.228330    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-model-cache\") pod \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") "
Sep 16 19:09:38.228463 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.228411    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-home\") pod \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") "
Sep 16 19:09:38.228463 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.228434    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkdxw\" (UniqueName: \"kubernetes.io/projected/a51c130b-11d9-4b77-99ea-3d152d1db2ea-kube-api-access-wkdxw\") pod \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") "
Sep 16 19:09:38.228463 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.228457    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-tmp-dir\") pod \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") "
Sep 16 19:09:38.228629 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.228474    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-dshm\") pod \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") "
Sep 16 19:09:38.228629 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.228492    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a51c130b-11d9-4b77-99ea-3d152d1db2ea-tls-certs\") pod \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\" (UID: \"a51c130b-11d9-4b77-99ea-3d152d1db2ea\") "
Sep 16 19:09:38.228629 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.228539    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "a51c130b-11d9-4b77-99ea-3d152d1db2ea" (UID: "a51c130b-11d9-4b77-99ea-3d152d1db2ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:38.228784 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.228651    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-home" (OuterVolumeSpecName: "home") pod "a51c130b-11d9-4b77-99ea-3d152d1db2ea" (UID: "a51c130b-11d9-4b77-99ea-3d152d1db2ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:38.228784 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.228688    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a51c130b-11d9-4b77-99ea-3d152d1db2ea" (UID: "a51c130b-11d9-4b77-99ea-3d152d1db2ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:38.228874 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.228851    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34684c6d-ecca-42fe-a6df-052c74193c88-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:09:38.228874 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.228871    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:09:38.228990 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.228884    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34684c6d-ecca-42fe-a6df-052c74193c88-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:09:38.228990 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.228898    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9fss\" (UniqueName: \"kubernetes.io/projected/34684c6d-ecca-42fe-a6df-052c74193c88-kube-api-access-k9fss\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:09:38.228990 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.228912    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:09:38.230545 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.230521    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-dshm" (OuterVolumeSpecName: "dshm") pod "a51c130b-11d9-4b77-99ea-3d152d1db2ea" (UID: "a51c130b-11d9-4b77-99ea-3d152d1db2ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:38.230629 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.230544    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a51c130b-11d9-4b77-99ea-3d152d1db2ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a51c130b-11d9-4b77-99ea-3d152d1db2ea" (UID: "a51c130b-11d9-4b77-99ea-3d152d1db2ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:38.230629 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.230588    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51c130b-11d9-4b77-99ea-3d152d1db2ea-kube-api-access-wkdxw" (OuterVolumeSpecName: "kube-api-access-wkdxw") pod "a51c130b-11d9-4b77-99ea-3d152d1db2ea" (UID: "a51c130b-11d9-4b77-99ea-3d152d1db2ea"). InnerVolumeSpecName "kube-api-access-wkdxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:38.330085 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.330036    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkdxw\" (UniqueName: \"kubernetes.io/projected/a51c130b-11d9-4b77-99ea-3d152d1db2ea-kube-api-access-wkdxw\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:09:38.330085 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.330057    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:09:38.330085 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.330068    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a51c130b-11d9-4b77-99ea-3d152d1db2ea-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:09:38.330085 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:38.330086    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a51c130b-11d9-4b77-99ea-3d152d1db2ea-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:09:39.016684 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:39.016620    2565 generic.go:356] "Generic (PLEG): container finished" podID="a51c130b-11d9-4b77-99ea-3d152d1db2ea" containerID="dcfa080affce6d31dfdca2ad2214034b6b5fbc4947ab4653ee98e22cb85c0d81" exitCode=0
Sep 16 19:09:39.016684 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:39.016668    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"
Sep 16 19:09:39.017147 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:39.016687    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c" event={"ID":"a51c130b-11d9-4b77-99ea-3d152d1db2ea","Type":"ContainerDied","Data":"dcfa080affce6d31dfdca2ad2214034b6b5fbc4947ab4653ee98e22cb85c0d81"}
Sep 16 19:09:39.017147 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:39.016733    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c" event={"ID":"a51c130b-11d9-4b77-99ea-3d152d1db2ea","Type":"ContainerDied","Data":"24f9e41f0f74234483f39aaa980d0f5738f7414b310599c7803b0af69db237ee"}
Sep 16 19:09:39.017147 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:39.016766    2565 scope.go:117] "RemoveContainer" containerID="dcfa080affce6d31dfdca2ad2214034b6b5fbc4947ab4653ee98e22cb85c0d81"
Sep 16 19:09:39.018175 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:39.018155    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"
Sep 16 19:09:39.018259 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:39.018189    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr" event={"ID":"34684c6d-ecca-42fe-a6df-052c74193c88","Type":"ContainerDied","Data":"2953fadaf7f058d47849819aeebf2a4fa0ea0d978a3cf2e91697b536cf6f1a8e"}
Sep 16 19:09:39.026503 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:39.026480    2565 scope.go:117] "RemoveContainer" containerID="dcfa080affce6d31dfdca2ad2214034b6b5fbc4947ab4653ee98e22cb85c0d81"
Sep 16 19:09:39.026787 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:09:39.026767    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcfa080affce6d31dfdca2ad2214034b6b5fbc4947ab4653ee98e22cb85c0d81\": container with ID starting with dcfa080affce6d31dfdca2ad2214034b6b5fbc4947ab4653ee98e22cb85c0d81 not found: ID does not exist" containerID="dcfa080affce6d31dfdca2ad2214034b6b5fbc4947ab4653ee98e22cb85c0d81"
Sep 16 19:09:39.026840 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:39.026794    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfa080affce6d31dfdca2ad2214034b6b5fbc4947ab4653ee98e22cb85c0d81"} err="failed to get container status \"dcfa080affce6d31dfdca2ad2214034b6b5fbc4947ab4653ee98e22cb85c0d81\": rpc error: code = NotFound desc = could not find container \"dcfa080affce6d31dfdca2ad2214034b6b5fbc4947ab4653ee98e22cb85c0d81\": container with ID starting with dcfa080affce6d31dfdca2ad2214034b6b5fbc4947ab4653ee98e22cb85c0d81 not found: ID does not exist"
Sep 16 19:09:39.026840 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:39.026811    2565 scope.go:117] "RemoveContainer" containerID="7ff6fb1c5c238a7dd6763468ec59078291544191c81d1f2d8848b3b428d3dca0"
Sep 16 19:09:39.040542 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:39.040522    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"]
Sep 16 19:09:39.053891 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:39.051208    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-jr8cr"]
Sep 16 19:09:39.067485 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:39.067461    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"]
Sep 16 19:09:39.069612 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:39.069593    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-nx25c"]
Sep 16 19:09:40.057293 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.056472    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"]
Sep 16 19:09:40.057293 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.057110    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51c130b-11d9-4b77-99ea-3d152d1db2ea" containerName="main"
Sep 16 19:09:40.057293 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.057130    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51c130b-11d9-4b77-99ea-3d152d1db2ea" containerName="main"
Sep 16 19:09:40.057293 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.057162    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34684c6d-ecca-42fe-a6df-052c74193c88" containerName="main"
Sep 16 19:09:40.057293 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.057171    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="34684c6d-ecca-42fe-a6df-052c74193c88" containerName="main"
Sep 16 19:09:40.057293 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.057283    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="34684c6d-ecca-42fe-a6df-052c74193c88" containerName="main"
Sep 16 19:09:40.057293 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.057294    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a51c130b-11d9-4b77-99ea-3d152d1db2ea" containerName="main"
Sep 16 19:09:40.062183 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.062161    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.066271 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.066249    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-dsvxd\""
Sep 16 19:09:40.066726 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.066712    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 16 19:09:40.067496 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.067482    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\""
Sep 16 19:09:40.067568 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.067535    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 16 19:09:40.085339 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.085313    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"]
Sep 16 19:09:40.144524 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.144486    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469bfb7c-b277-4200-a915-52f90fb7c9ba-tls-certs\") pod \"stop-v2-kserve-7f68747c45-2jp5r\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.144524 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.144526    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-dshm\") pod \"stop-v2-kserve-7f68747c45-2jp5r\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.144733 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.144542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-model-cache\") pod \"stop-v2-kserve-7f68747c45-2jp5r\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.144733 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.144601    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-2jp5r\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.144733 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.144689    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-home\") pod \"stop-v2-kserve-7f68747c45-2jp5r\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.144733 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.144714    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq2hp\" (UniqueName: \"kubernetes.io/projected/469bfb7c-b277-4200-a915-52f90fb7c9ba-kube-api-access-dq2hp\") pod \"stop-v2-kserve-7f68747c45-2jp5r\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.207066 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.207036    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34684c6d-ecca-42fe-a6df-052c74193c88" path="/var/lib/kubelet/pods/34684c6d-ecca-42fe-a6df-052c74193c88/volumes"
Sep 16 19:09:40.207471 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.207457    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a51c130b-11d9-4b77-99ea-3d152d1db2ea" path="/var/lib/kubelet/pods/a51c130b-11d9-4b77-99ea-3d152d1db2ea/volumes"
Sep 16 19:09:40.245984 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.245959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469bfb7c-b277-4200-a915-52f90fb7c9ba-tls-certs\") pod \"stop-v2-kserve-7f68747c45-2jp5r\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.246119 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.245988    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-dshm\") pod \"stop-v2-kserve-7f68747c45-2jp5r\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.246119 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.246004    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-model-cache\") pod \"stop-v2-kserve-7f68747c45-2jp5r\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.246119 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.246024    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-2jp5r\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.246119 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.246066    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-home\") pod \"stop-v2-kserve-7f68747c45-2jp5r\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.246119 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.246088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq2hp\" (UniqueName: \"kubernetes.io/projected/469bfb7c-b277-4200-a915-52f90fb7c9ba-kube-api-access-dq2hp\") pod \"stop-v2-kserve-7f68747c45-2jp5r\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.246606 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.246567    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-model-cache\") pod \"stop-v2-kserve-7f68747c45-2jp5r\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.246704 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.246578    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-2jp5r\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.246704 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.246612    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-home\") pod \"stop-v2-kserve-7f68747c45-2jp5r\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.248432 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.248402    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-dshm\") pod \"stop-v2-kserve-7f68747c45-2jp5r\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.248634 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.248613    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469bfb7c-b277-4200-a915-52f90fb7c9ba-tls-certs\") pod \"stop-v2-kserve-7f68747c45-2jp5r\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.258546 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.258525    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq2hp\" (UniqueName: \"kubernetes.io/projected/469bfb7c-b277-4200-a915-52f90fb7c9ba-kube-api-access-dq2hp\") pod \"stop-v2-kserve-7f68747c45-2jp5r\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.373667 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.373568    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:40.657697 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:40.657663    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"]
Sep 16 19:09:40.659950 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:09:40.659926    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469bfb7c_b277_4200_a915_52f90fb7c9ba.slice/crio-c6f6160b102e651af1b2c785043c61d1b0be13616b54bd215550fd739342bcb3 WatchSource:0}: Error finding container c6f6160b102e651af1b2c785043c61d1b0be13616b54bd215550fd739342bcb3: Status 404 returned error can't find the container with id c6f6160b102e651af1b2c785043c61d1b0be13616b54bd215550fd739342bcb3
Sep 16 19:09:41.032581 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:41.032546    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r" event={"ID":"469bfb7c-b277-4200-a915-52f90fb7c9ba","Type":"ContainerStarted","Data":"20db1331d50754c9b9aa1ebbe1d185534e1690f9c54f77a4daf2450578c70c30"}
Sep 16 19:09:41.032739 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:41.032604    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r" event={"ID":"469bfb7c-b277-4200-a915-52f90fb7c9ba","Type":"ContainerStarted","Data":"c6f6160b102e651af1b2c785043c61d1b0be13616b54bd215550fd739342bcb3"}
Sep 16 19:09:41.056033 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:41.055983    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r" podStartSLOduration=1.055968908 podStartE2EDuration="1.055968908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:41.055859886 +0000 UTC m=+1819.415377881" watchObservedRunningTime="2026-09-16 19:09:41.055968908 +0000 UTC m=+1819.415486902"
Sep 16 19:09:41.374292 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:41.374214    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:50.373882 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:50.373852    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:50.387699 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:50.387675    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:09:51.097039 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:09:51.097004    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:10:26.194102 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:26.194060    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"]
Sep 16 19:10:41.216291 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.216227    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r" podUID="469bfb7c-b277-4200-a915-52f90fb7c9ba" containerName="main" containerID="cri-o://20db1331d50754c9b9aa1ebbe1d185534e1690f9c54f77a4daf2450578c70c30" gracePeriod=45
Sep 16 19:10:41.355157 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.355122    2565 generic.go:356] "Generic (PLEG): container finished" podID="469bfb7c-b277-4200-a915-52f90fb7c9ba" containerID="20db1331d50754c9b9aa1ebbe1d185534e1690f9c54f77a4daf2450578c70c30" exitCode=0
Sep 16 19:10:41.355286 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.355192    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r" event={"ID":"469bfb7c-b277-4200-a915-52f90fb7c9ba","Type":"ContainerDied","Data":"20db1331d50754c9b9aa1ebbe1d185534e1690f9c54f77a4daf2450578c70c30"}
Sep 16 19:10:41.468674 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.468623    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:10:41.626552 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.626521    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-model-cache\") pod \"469bfb7c-b277-4200-a915-52f90fb7c9ba\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") "
Sep 16 19:10:41.626676 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.626561    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-home\") pod \"469bfb7c-b277-4200-a915-52f90fb7c9ba\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") "
Sep 16 19:10:41.626676 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.626594    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-tmp-dir\") pod \"469bfb7c-b277-4200-a915-52f90fb7c9ba\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") "
Sep 16 19:10:41.626676 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.626616    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469bfb7c-b277-4200-a915-52f90fb7c9ba-tls-certs\") pod \"469bfb7c-b277-4200-a915-52f90fb7c9ba\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") "
Sep 16 19:10:41.626676 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.626635    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq2hp\" (UniqueName: \"kubernetes.io/projected/469bfb7c-b277-4200-a915-52f90fb7c9ba-kube-api-access-dq2hp\") pod \"469bfb7c-b277-4200-a915-52f90fb7c9ba\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") "
Sep 16 19:10:41.626676 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.626675    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-dshm\") pod \"469bfb7c-b277-4200-a915-52f90fb7c9ba\" (UID: \"469bfb7c-b277-4200-a915-52f90fb7c9ba\") "
Sep 16 19:10:41.626947 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.626806    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "469bfb7c-b277-4200-a915-52f90fb7c9ba" (UID: "469bfb7c-b277-4200-a915-52f90fb7c9ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:41.626947 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.626890    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "469bfb7c-b277-4200-a915-52f90fb7c9ba" (UID: "469bfb7c-b277-4200-a915-52f90fb7c9ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:41.626947 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.626928    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:10:41.627071 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.626954    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-home" (OuterVolumeSpecName: "home") pod "469bfb7c-b277-4200-a915-52f90fb7c9ba" (UID: "469bfb7c-b277-4200-a915-52f90fb7c9ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:41.628789 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.628759    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469bfb7c-b277-4200-a915-52f90fb7c9ba-kube-api-access-dq2hp" (OuterVolumeSpecName: "kube-api-access-dq2hp") pod "469bfb7c-b277-4200-a915-52f90fb7c9ba" (UID: "469bfb7c-b277-4200-a915-52f90fb7c9ba"). InnerVolumeSpecName "kube-api-access-dq2hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:41.628902 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.628821    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/469bfb7c-b277-4200-a915-52f90fb7c9ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "469bfb7c-b277-4200-a915-52f90fb7c9ba" (UID: "469bfb7c-b277-4200-a915-52f90fb7c9ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:41.628902 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.628824    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-dshm" (OuterVolumeSpecName: "dshm") pod "469bfb7c-b277-4200-a915-52f90fb7c9ba" (UID: "469bfb7c-b277-4200-a915-52f90fb7c9ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:41.728150 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.728128    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:10:41.728150 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.728149    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:10:41.728286 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.728159    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469bfb7c-b277-4200-a915-52f90fb7c9ba-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:10:41.728286 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.728168    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq2hp\" (UniqueName: \"kubernetes.io/projected/469bfb7c-b277-4200-a915-52f90fb7c9ba-kube-api-access-dq2hp\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:10:41.728286 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:41.728177    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469bfb7c-b277-4200-a915-52f90fb7c9ba-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:10:42.362783 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:42.362751    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"
Sep 16 19:10:42.363216 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:42.362787    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r" event={"ID":"469bfb7c-b277-4200-a915-52f90fb7c9ba","Type":"ContainerDied","Data":"c6f6160b102e651af1b2c785043c61d1b0be13616b54bd215550fd739342bcb3"}
Sep 16 19:10:42.363216 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:42.362844    2565 scope.go:117] "RemoveContainer" containerID="20db1331d50754c9b9aa1ebbe1d185534e1690f9c54f77a4daf2450578c70c30"
Sep 16 19:10:42.403603 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:42.403572    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"]
Sep 16 19:10:42.414240 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:42.414215    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-2jp5r"]
Sep 16 19:10:44.208201 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:10:44.208128    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="469bfb7c-b277-4200-a915-52f90fb7c9ba" path="/var/lib/kubelet/pods/469bfb7c-b277-4200-a915-52f90fb7c9ba/volumes"
Sep 16 19:11:03.592692 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.592651    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"]
Sep 16 19:11:03.593181 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.593164    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="469bfb7c-b277-4200-a915-52f90fb7c9ba" containerName="main"
Sep 16 19:11:03.593226 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.593184    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="469bfb7c-b277-4200-a915-52f90fb7c9ba" containerName="main"
Sep 16 19:11:03.593266 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.593250    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="469bfb7c-b277-4200-a915-52f90fb7c9ba" containerName="main"
Sep 16 19:11:03.596267 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.596246    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:03.608038 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.608011    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 16 19:11:03.608217 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.608011    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 16 19:11:03.608217 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.608065    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-tp2mk\""
Sep 16 19:11:03.608393 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.608082    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Sep 16 19:11:03.624520 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.624492    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"]
Sep 16 19:11:03.693507 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.693476    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9f2b\" (UniqueName: \"kubernetes.io/projected/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-kube-api-access-t9f2b\") pod \"decom-v1-kserve-6cb797fb75-8l68c\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:03.693507 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.693507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-model-cache\") pod \"decom-v1-kserve-6cb797fb75-8l68c\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:03.693793 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.693538    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-home\") pod \"decom-v1-kserve-6cb797fb75-8l68c\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:03.693793 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.693649    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-dshm\") pod \"decom-v1-kserve-6cb797fb75-8l68c\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:03.693793 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.693690    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-8l68c\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:03.693793 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.693717    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-8l68c\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:03.794208 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.794179    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-home\") pod \"decom-v1-kserve-6cb797fb75-8l68c\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:03.794327 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.794220    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-dshm\") pod \"decom-v1-kserve-6cb797fb75-8l68c\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:03.794327 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.794241    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-8l68c\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:03.794327 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.794270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-8l68c\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:03.794327 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.794315    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9f2b\" (UniqueName: \"kubernetes.io/projected/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-kube-api-access-t9f2b\") pod \"decom-v1-kserve-6cb797fb75-8l68c\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:03.794583 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.794368    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-model-cache\") pod \"decom-v1-kserve-6cb797fb75-8l68c\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:03.794647 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.794596    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-home\") pod \"decom-v1-kserve-6cb797fb75-8l68c\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:03.794711 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.794698    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-model-cache\") pod \"decom-v1-kserve-6cb797fb75-8l68c\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:03.794793 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.794771    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-8l68c\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:03.796607 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.796584    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-dshm\") pod \"decom-v1-kserve-6cb797fb75-8l68c\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:03.796811 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.796790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-8l68c\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:03.802663 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.802642    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9f2b\" (UniqueName: \"kubernetes.io/projected/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-kube-api-access-t9f2b\") pod \"decom-v1-kserve-6cb797fb75-8l68c\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:03.909949 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:03.909896    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:04.191066 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:04.191036    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"]
Sep 16 19:11:04.192235 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:11:04.192207    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79da9b9a_73d5_47cf_b8b7_dd33ac8d3cb3.slice/crio-1e54ca28381c73dc132a82101696d2f0dcd4508049731f0012c20f17a4bf64b6 WatchSource:0}: Error finding container 1e54ca28381c73dc132a82101696d2f0dcd4508049731f0012c20f17a4bf64b6: Status 404 returned error can't find the container with id 1e54ca28381c73dc132a82101696d2f0dcd4508049731f0012c20f17a4bf64b6
Sep 16 19:11:04.486761 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:04.486721    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c" event={"ID":"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3","Type":"ContainerStarted","Data":"0547dd54a35c69200f57bc65541ab87434ffd51380138cb5b34d5ef8b2dc86a2"}
Sep 16 19:11:04.486949 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:04.486786    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c" event={"ID":"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3","Type":"ContainerStarted","Data":"1e54ca28381c73dc132a82101696d2f0dcd4508049731f0012c20f17a4bf64b6"}
Sep 16 19:11:04.514112 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:04.513856    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c" podStartSLOduration=1.5138365409999999 podStartE2EDuration="1.513836541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:04.512627441 +0000 UTC m=+1902.872145471" watchObservedRunningTime="2026-09-16 19:11:04.513836541 +0000 UTC m=+1902.873354537"
Sep 16 19:11:04.910420 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:04.910325    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:13.910540 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:13.910503    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:13.922730 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:13.922708    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:11:14.553228 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:11:14.553194    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:12:06.828112 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:06.828078    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"]
Sep 16 19:12:12.777053 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:12.777017    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"]
Sep 16 19:12:12.780701 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:12.780676    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:12.785170 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:12.785150    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 16 19:12:12.786141 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:12.786100    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Sep 16 19:12:12.786300 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:12.786190    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-sslmm\""
Sep 16 19:12:12.786418 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:12.786195    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 16 19:12:12.804375 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:12.804332    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"]
Sep 16 19:12:12.931569 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:12.931536    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-home\") pod \"leave-v1-kserve-65684d4f5-f6wwc\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:12.931569 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:12.931571    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/621ed769-6e64-45a6-878f-5f99fff5e147-tls-certs\") pod \"leave-v1-kserve-65684d4f5-f6wwc\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:12.931766 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:12.931595    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-f6wwc\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:12.931766 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:12.931638    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s2bp\" (UniqueName: \"kubernetes.io/projected/621ed769-6e64-45a6-878f-5f99fff5e147-kube-api-access-7s2bp\") pod \"leave-v1-kserve-65684d4f5-f6wwc\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:12.931766 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:12.931674    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-model-cache\") pod \"leave-v1-kserve-65684d4f5-f6wwc\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:12.931766 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:12.931691    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-dshm\") pod \"leave-v1-kserve-65684d4f5-f6wwc\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:13.032611 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:13.032539    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-model-cache\") pod \"leave-v1-kserve-65684d4f5-f6wwc\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:13.032611 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:13.032574    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-dshm\") pod \"leave-v1-kserve-65684d4f5-f6wwc\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:13.032815 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:13.032635    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-home\") pod \"leave-v1-kserve-65684d4f5-f6wwc\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:13.032815 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:13.032661    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/621ed769-6e64-45a6-878f-5f99fff5e147-tls-certs\") pod \"leave-v1-kserve-65684d4f5-f6wwc\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:13.032815 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:13.032685    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-f6wwc\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:13.032815 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:13.032714    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s2bp\" (UniqueName: \"kubernetes.io/projected/621ed769-6e64-45a6-878f-5f99fff5e147-kube-api-access-7s2bp\") pod \"leave-v1-kserve-65684d4f5-f6wwc\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:13.033055 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:13.033032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-home\") pod \"leave-v1-kserve-65684d4f5-f6wwc\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:13.033099 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:13.033070    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-model-cache\") pod \"leave-v1-kserve-65684d4f5-f6wwc\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:13.033135 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:13.033100    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-f6wwc\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:13.035083 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:13.035054    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-dshm\") pod \"leave-v1-kserve-65684d4f5-f6wwc\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:13.035187 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:13.035119    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/621ed769-6e64-45a6-878f-5f99fff5e147-tls-certs\") pod \"leave-v1-kserve-65684d4f5-f6wwc\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:13.043713 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:13.043691    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s2bp\" (UniqueName: \"kubernetes.io/projected/621ed769-6e64-45a6-878f-5f99fff5e147-kube-api-access-7s2bp\") pod \"leave-v1-kserve-65684d4f5-f6wwc\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:13.091038 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:13.091014    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:13.368378 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:13.368328    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"]
Sep 16 19:12:13.370155 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:12:13.370128    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621ed769_6e64_45a6_878f_5f99fff5e147.slice/crio-8fa8b86712af4358212d8154de6a858fbfa47886738edbc0a361227ba6bdc108 WatchSource:0}: Error finding container 8fa8b86712af4358212d8154de6a858fbfa47886738edbc0a361227ba6bdc108: Status 404 returned error can't find the container with id 8fa8b86712af4358212d8154de6a858fbfa47886738edbc0a361227ba6bdc108
Sep 16 19:12:13.863290 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:13.863256    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc" event={"ID":"621ed769-6e64-45a6-878f-5f99fff5e147","Type":"ContainerStarted","Data":"2dc3474db46bc872cd65763ffbfa6b7285efe66044d4e984b86f600c886925aa"}
Sep 16 19:12:13.863766 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:13.863311    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc" event={"ID":"621ed769-6e64-45a6-878f-5f99fff5e147","Type":"ContainerStarted","Data":"8fa8b86712af4358212d8154de6a858fbfa47886738edbc0a361227ba6bdc108"}
Sep 16 19:12:13.888363 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:13.888300    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc" podStartSLOduration=1.888285099 podStartE2EDuration="1.888285099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:13.886757518 +0000 UTC m=+1972.246275512" watchObservedRunningTime="2026-09-16 19:12:13.888285099 +0000 UTC m=+1972.247803110"
Sep 16 19:12:14.092087 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:14.092020    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:18.753332 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.753296    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"]
Sep 16 19:12:18.757471 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.757446    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:18.762055 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.762030    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Sep 16 19:12:18.768393 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.768370    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"]
Sep 16 19:12:18.880365 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.880319    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt6hf\" (UniqueName: \"kubernetes.io/projected/55ceecfd-22bb-4683-94da-fb0077d5a22f-kube-api-access-lt6hf\") pod \"leave-v2-kserve-556b4dddf9-bc86w\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:18.880506 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.880379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-bc86w\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:18.880506 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.880438    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ceecfd-22bb-4683-94da-fb0077d5a22f-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-bc86w\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:18.880506 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.880477    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-model-cache\") pod \"leave-v2-kserve-556b4dddf9-bc86w\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:18.880659 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.880507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-dshm\") pod \"leave-v2-kserve-556b4dddf9-bc86w\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:18.880659 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.880626    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-home\") pod \"leave-v2-kserve-556b4dddf9-bc86w\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:18.981011 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.980981    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-home\") pod \"leave-v2-kserve-556b4dddf9-bc86w\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:18.981159 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.981019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt6hf\" (UniqueName: \"kubernetes.io/projected/55ceecfd-22bb-4683-94da-fb0077d5a22f-kube-api-access-lt6hf\") pod \"leave-v2-kserve-556b4dddf9-bc86w\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:18.981159 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.981035    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-bc86w\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:18.981159 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.981059    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ceecfd-22bb-4683-94da-fb0077d5a22f-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-bc86w\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:18.981159 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.981094    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-model-cache\") pod \"leave-v2-kserve-556b4dddf9-bc86w\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:18.981159 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.981127    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-dshm\") pod \"leave-v2-kserve-556b4dddf9-bc86w\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:18.981444 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.981327    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-home\") pod \"leave-v2-kserve-556b4dddf9-bc86w\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:18.981444 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.981429    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-bc86w\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:18.981563 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.981529    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-model-cache\") pod \"leave-v2-kserve-556b4dddf9-bc86w\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:18.983476 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.983453    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-dshm\") pod \"leave-v2-kserve-556b4dddf9-bc86w\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:18.983710 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.983691    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ceecfd-22bb-4683-94da-fb0077d5a22f-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-bc86w\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:18.990120 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:18.990091    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt6hf\" (UniqueName: \"kubernetes.io/projected/55ceecfd-22bb-4683-94da-fb0077d5a22f-kube-api-access-lt6hf\") pod \"leave-v2-kserve-556b4dddf9-bc86w\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:19.068340 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:19.068282    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:19.346406 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:19.346381    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"]
Sep 16 19:12:19.348159 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:12:19.348129    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ceecfd_22bb_4683_94da_fb0077d5a22f.slice/crio-b94f3b75ab3b91e1cd3866fe026a28374a0a594dbe5f9cda4bde90f8b51093d0 WatchSource:0}: Error finding container b94f3b75ab3b91e1cd3866fe026a28374a0a594dbe5f9cda4bde90f8b51093d0: Status 404 returned error can't find the container with id b94f3b75ab3b91e1cd3866fe026a28374a0a594dbe5f9cda4bde90f8b51093d0
Sep 16 19:12:19.907789 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:19.907747    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w" event={"ID":"55ceecfd-22bb-4683-94da-fb0077d5a22f","Type":"ContainerStarted","Data":"cd0c1e771f5d5c958ececee3966bbf73f7914143e8c21b44288f40618d9c92c6"}
Sep 16 19:12:19.908161 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:19.907830    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w" event={"ID":"55ceecfd-22bb-4683-94da-fb0077d5a22f","Type":"ContainerStarted","Data":"b94f3b75ab3b91e1cd3866fe026a28374a0a594dbe5f9cda4bde90f8b51093d0"}
Sep 16 19:12:19.931724 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:19.931674    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w" podStartSLOduration=1.931660356 podStartE2EDuration="1.931660356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:19.928958976 +0000 UTC m=+1978.288476962" watchObservedRunningTime="2026-09-16 19:12:19.931660356 +0000 UTC m=+1978.291178351"
Sep 16 19:12:20.069135 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:20.069104    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:21.846670 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:21.846624    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c" podUID="79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3" containerName="main" containerID="cri-o://0547dd54a35c69200f57bc65541ab87434ffd51380138cb5b34d5ef8b2dc86a2" gracePeriod=45
Sep 16 19:12:22.113538 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.113515    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:12:22.308801 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.308775    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-model-cache\") pod \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") "
Sep 16 19:12:22.308938 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.308832    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-tmp-dir\") pod \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") "
Sep 16 19:12:22.308938 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.308856    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9f2b\" (UniqueName: \"kubernetes.io/projected/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-kube-api-access-t9f2b\") pod \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") "
Sep 16 19:12:22.308938 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.308889    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-tls-certs\") pod \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") "
Sep 16 19:12:22.308938 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.308904    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-home\") pod \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") "
Sep 16 19:12:22.309228 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.308949    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-dshm\") pod \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\" (UID: \"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3\") "
Sep 16 19:12:22.309228 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.309041    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3" (UID: "79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:22.309228 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.309076    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-model-cache" (OuterVolumeSpecName: "model-cache") pod "79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3" (UID: "79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:22.309228 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.309193    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-home" (OuterVolumeSpecName: "home") pod "79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3" (UID: "79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:22.309228 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.309195    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:12:22.309228 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.309226    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:12:22.311134 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.311069    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-kube-api-access-t9f2b" (OuterVolumeSpecName: "kube-api-access-t9f2b") pod "79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3" (UID: "79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3"). InnerVolumeSpecName "kube-api-access-t9f2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:22.311134 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.311127    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3" (UID: "79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:22.311309 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.311177    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-dshm" (OuterVolumeSpecName: "dshm") pod "79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3" (UID: "79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:22.410450 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.410397    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9f2b\" (UniqueName: \"kubernetes.io/projected/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-kube-api-access-t9f2b\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:12:22.410450 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.410419    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:12:22.410450 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.410428    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:12:22.410450 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.410435    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:12:22.926455 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.926425    2565 generic.go:356] "Generic (PLEG): container finished" podID="79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3" containerID="0547dd54a35c69200f57bc65541ab87434ffd51380138cb5b34d5ef8b2dc86a2" exitCode=0
Sep 16 19:12:22.926824 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.926472    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"
Sep 16 19:12:22.926824 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.926507    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c" event={"ID":"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3","Type":"ContainerDied","Data":"0547dd54a35c69200f57bc65541ab87434ffd51380138cb5b34d5ef8b2dc86a2"}
Sep 16 19:12:22.926824 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.926562    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c" event={"ID":"79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3","Type":"ContainerDied","Data":"1e54ca28381c73dc132a82101696d2f0dcd4508049731f0012c20f17a4bf64b6"}
Sep 16 19:12:22.926824 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.926585    2565 scope.go:117] "RemoveContainer" containerID="0547dd54a35c69200f57bc65541ab87434ffd51380138cb5b34d5ef8b2dc86a2"
Sep 16 19:12:22.936147 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.936131    2565 scope.go:117] "RemoveContainer" containerID="0547dd54a35c69200f57bc65541ab87434ffd51380138cb5b34d5ef8b2dc86a2"
Sep 16 19:12:22.936441 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:12:22.936418    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0547dd54a35c69200f57bc65541ab87434ffd51380138cb5b34d5ef8b2dc86a2\": container with ID starting with 0547dd54a35c69200f57bc65541ab87434ffd51380138cb5b34d5ef8b2dc86a2 not found: ID does not exist" containerID="0547dd54a35c69200f57bc65541ab87434ffd51380138cb5b34d5ef8b2dc86a2"
Sep 16 19:12:22.936531 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.936448    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0547dd54a35c69200f57bc65541ab87434ffd51380138cb5b34d5ef8b2dc86a2"} err="failed to get container status \"0547dd54a35c69200f57bc65541ab87434ffd51380138cb5b34d5ef8b2dc86a2\": rpc error: code = NotFound desc = could not find container \"0547dd54a35c69200f57bc65541ab87434ffd51380138cb5b34d5ef8b2dc86a2\": container with ID starting with 0547dd54a35c69200f57bc65541ab87434ffd51380138cb5b34d5ef8b2dc86a2 not found: ID does not exist"
Sep 16 19:12:22.949897 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.949876    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"]
Sep 16 19:12:22.955895 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:22.954606    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8l68c"]
Sep 16 19:12:23.091739 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:23.091716    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:23.104646 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:23.104625    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:23.944436 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:23.944402    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:12:24.207051 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:24.206974    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3" path="/var/lib/kubelet/pods/79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3/volumes"
Sep 16 19:12:29.069420 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:29.069383    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:29.081502 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:29.081478    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:12:29.974971 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:12:29.974946    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:13:01.781600 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:01.781564    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"]
Sep 16 19:13:02.662853 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:02.662818    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"]
Sep 16 19:13:08.447683 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.447648    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"]
Sep 16 19:13:08.448157 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.448143    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3" containerName="main"
Sep 16 19:13:08.448226 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.448164    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3" containerName="main"
Sep 16 19:13:08.448293 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.448252    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="79da9b9a-73d5-47cf-b8b7-dd33ac8d3cb3" containerName="main"
Sep 16 19:13:08.451433 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.451410    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:08.454757 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.454716    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 16 19:13:08.454990 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.454716    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 16 19:13:08.455103 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.454722    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-4vgc2\""
Sep 16 19:13:08.455103 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.454796    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\""
Sep 16 19:13:08.461889 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.461865    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"]
Sep 16 19:13:08.568526 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.568489    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-4zhhq\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:08.568683 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.568578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-dshm\") pod \"tri-v1-kserve-79f8f98bc5-4zhhq\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:08.568683 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.568607    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-4zhhq\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:08.568683 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.568647    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h52dk\" (UniqueName: \"kubernetes.io/projected/c2612890-d754-4c96-b63e-e5353b41dc0c-kube-api-access-h52dk\") pod \"tri-v1-kserve-79f8f98bc5-4zhhq\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:08.568809 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.568693    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2612890-d754-4c96-b63e-e5353b41dc0c-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-4zhhq\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:08.568809 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.568781    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-home\") pod \"tri-v1-kserve-79f8f98bc5-4zhhq\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:08.670071 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.670035    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-home\") pod \"tri-v1-kserve-79f8f98bc5-4zhhq\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:08.670213 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.670082    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-4zhhq\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:08.670213 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.670197    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-dshm\") pod \"tri-v1-kserve-79f8f98bc5-4zhhq\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:08.670305 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.670234    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-4zhhq\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:08.670404 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.670383    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h52dk\" (UniqueName: \"kubernetes.io/projected/c2612890-d754-4c96-b63e-e5353b41dc0c-kube-api-access-h52dk\") pod \"tri-v1-kserve-79f8f98bc5-4zhhq\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:08.670472 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.670451    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2612890-d754-4c96-b63e-e5353b41dc0c-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-4zhhq\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:08.670527 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.670494    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-4zhhq\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:08.670578 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.670535    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-home\") pod \"tri-v1-kserve-79f8f98bc5-4zhhq\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:08.670578 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.670556    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-4zhhq\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:08.672443 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.672415    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-dshm\") pod \"tri-v1-kserve-79f8f98bc5-4zhhq\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:08.672733 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.672715    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2612890-d754-4c96-b63e-e5353b41dc0c-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-4zhhq\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:08.678113 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.678092    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h52dk\" (UniqueName: \"kubernetes.io/projected/c2612890-d754-4c96-b63e-e5353b41dc0c-kube-api-access-h52dk\") pod \"tri-v1-kserve-79f8f98bc5-4zhhq\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:08.763188 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:08.763163    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:09.040943 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:09.040916    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"]
Sep 16 19:13:09.042533 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:13:09.042500    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2612890_d754_4c96_b63e_e5353b41dc0c.slice/crio-cfe5ac2ccd1d7196f4945b6550cee9e451909d52cbfb8426028e372d08790c87 WatchSource:0}: Error finding container cfe5ac2ccd1d7196f4945b6550cee9e451909d52cbfb8426028e372d08790c87: Status 404 returned error can't find the container with id cfe5ac2ccd1d7196f4945b6550cee9e451909d52cbfb8426028e372d08790c87
Sep 16 19:13:09.174738 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:09.174709    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq" event={"ID":"c2612890-d754-4c96-b63e-e5353b41dc0c","Type":"ContainerStarted","Data":"fc475342cd23f877c66f1ae85303a3d040ad480c83cc9b9b1092964835d5812d"}
Sep 16 19:13:09.174835 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:09.174760    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq" event={"ID":"c2612890-d754-4c96-b63e-e5353b41dc0c","Type":"ContainerStarted","Data":"cfe5ac2ccd1d7196f4945b6550cee9e451909d52cbfb8426028e372d08790c87"}
Sep 16 19:13:09.206328 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:09.206279    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq" podStartSLOduration=1.206264523 podStartE2EDuration="1.206264523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:09.203893506 +0000 UTC m=+2027.563411502" watchObservedRunningTime="2026-09-16 19:13:09.206264523 +0000 UTC m=+2027.565782518"
Sep 16 19:13:09.763771 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:09.763741    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:16.801811 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:16.801727    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc" podUID="621ed769-6e64-45a6-878f-5f99fff5e147" containerName="main" containerID="cri-o://2dc3474db46bc872cd65763ffbfa6b7285efe66044d4e984b86f600c886925aa" gracePeriod=45
Sep 16 19:13:16.933888 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:16.933851    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc" podUID="621ed769-6e64-45a6-878f-5f99fff5e147" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 16 19:13:17.060650 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.060596    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:13:17.142876 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.142848    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-home\") pod \"621ed769-6e64-45a6-878f-5f99fff5e147\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") "
Sep 16 19:13:17.142984 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.142918    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-model-cache\") pod \"621ed769-6e64-45a6-878f-5f99fff5e147\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") "
Sep 16 19:13:17.142984 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.142942    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-dshm\") pod \"621ed769-6e64-45a6-878f-5f99fff5e147\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") "
Sep 16 19:13:17.142984 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.142967    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-tmp-dir\") pod \"621ed769-6e64-45a6-878f-5f99fff5e147\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") "
Sep 16 19:13:17.143151 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.142986    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/621ed769-6e64-45a6-878f-5f99fff5e147-tls-certs\") pod \"621ed769-6e64-45a6-878f-5f99fff5e147\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") "
Sep 16 19:13:17.143151 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.143029    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s2bp\" (UniqueName: \"kubernetes.io/projected/621ed769-6e64-45a6-878f-5f99fff5e147-kube-api-access-7s2bp\") pod \"621ed769-6e64-45a6-878f-5f99fff5e147\" (UID: \"621ed769-6e64-45a6-878f-5f99fff5e147\") "
Sep 16 19:13:17.143151 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.143075    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-home" (OuterVolumeSpecName: "home") pod "621ed769-6e64-45a6-878f-5f99fff5e147" (UID: "621ed769-6e64-45a6-878f-5f99fff5e147"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:17.143312 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.143240    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "621ed769-6e64-45a6-878f-5f99fff5e147" (UID: "621ed769-6e64-45a6-878f-5f99fff5e147"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:17.143312 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.143259    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-model-cache" (OuterVolumeSpecName: "model-cache") pod "621ed769-6e64-45a6-878f-5f99fff5e147" (UID: "621ed769-6e64-45a6-878f-5f99fff5e147"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:17.143442 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.143368    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:13:17.143442 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.143390    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:13:17.143442 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.143405    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:13:17.145122 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.145098    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-dshm" (OuterVolumeSpecName: "dshm") pod "621ed769-6e64-45a6-878f-5f99fff5e147" (UID: "621ed769-6e64-45a6-878f-5f99fff5e147"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:17.145212 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.145148    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/621ed769-6e64-45a6-878f-5f99fff5e147-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "621ed769-6e64-45a6-878f-5f99fff5e147" (UID: "621ed769-6e64-45a6-878f-5f99fff5e147"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:17.145300 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.145270    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621ed769-6e64-45a6-878f-5f99fff5e147-kube-api-access-7s2bp" (OuterVolumeSpecName: "kube-api-access-7s2bp") pod "621ed769-6e64-45a6-878f-5f99fff5e147" (UID: "621ed769-6e64-45a6-878f-5f99fff5e147"). InnerVolumeSpecName "kube-api-access-7s2bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:17.220402 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.220375    2565 generic.go:356] "Generic (PLEG): container finished" podID="621ed769-6e64-45a6-878f-5f99fff5e147" containerID="2dc3474db46bc872cd65763ffbfa6b7285efe66044d4e984b86f600c886925aa" exitCode=0
Sep 16 19:13:17.220512 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.220421    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"
Sep 16 19:13:17.220512 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.220462    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc" event={"ID":"621ed769-6e64-45a6-878f-5f99fff5e147","Type":"ContainerDied","Data":"2dc3474db46bc872cd65763ffbfa6b7285efe66044d4e984b86f600c886925aa"}
Sep 16 19:13:17.220586 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.220514    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc" event={"ID":"621ed769-6e64-45a6-878f-5f99fff5e147","Type":"ContainerDied","Data":"8fa8b86712af4358212d8154de6a858fbfa47886738edbc0a361227ba6bdc108"}
Sep 16 19:13:17.220586 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.220543    2565 scope.go:117] "RemoveContainer" containerID="2dc3474db46bc872cd65763ffbfa6b7285efe66044d4e984b86f600c886925aa"
Sep 16 19:13:17.229487 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.229470    2565 scope.go:117] "RemoveContainer" containerID="2dc3474db46bc872cd65763ffbfa6b7285efe66044d4e984b86f600c886925aa"
Sep 16 19:13:17.229757 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:13:17.229741    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc3474db46bc872cd65763ffbfa6b7285efe66044d4e984b86f600c886925aa\": container with ID starting with 2dc3474db46bc872cd65763ffbfa6b7285efe66044d4e984b86f600c886925aa not found: ID does not exist" containerID="2dc3474db46bc872cd65763ffbfa6b7285efe66044d4e984b86f600c886925aa"
Sep 16 19:13:17.229808 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.229764    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc3474db46bc872cd65763ffbfa6b7285efe66044d4e984b86f600c886925aa"} err="failed to get container status \"2dc3474db46bc872cd65763ffbfa6b7285efe66044d4e984b86f600c886925aa\": rpc error: code = NotFound desc = could not find container \"2dc3474db46bc872cd65763ffbfa6b7285efe66044d4e984b86f600c886925aa\": container with ID starting with 2dc3474db46bc872cd65763ffbfa6b7285efe66044d4e984b86f600c886925aa not found: ID does not exist"
Sep 16 19:13:17.244182 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.244165    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/621ed769-6e64-45a6-878f-5f99fff5e147-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:13:17.244182 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.244182    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/621ed769-6e64-45a6-878f-5f99fff5e147-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:13:17.244303 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.244193    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s2bp\" (UniqueName: \"kubernetes.io/projected/621ed769-6e64-45a6-878f-5f99fff5e147-kube-api-access-7s2bp\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:13:17.257026 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.257000    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"]
Sep 16 19:13:17.264000 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.263981    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-f6wwc"]
Sep 16 19:13:17.683843 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.683811    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w" podUID="55ceecfd-22bb-4683-94da-fb0077d5a22f" containerName="main" containerID="cri-o://cd0c1e771f5d5c958ececee3966bbf73f7914143e8c21b44288f40618d9c92c6" gracePeriod=45
Sep 16 19:13:17.939773 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:17.939715    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:13:18.049591 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.049564    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-dshm\") pod \"55ceecfd-22bb-4683-94da-fb0077d5a22f\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") "
Sep 16 19:13:18.049719 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.049596    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-home\") pod \"55ceecfd-22bb-4683-94da-fb0077d5a22f\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") "
Sep 16 19:13:18.049719 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.049642    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-model-cache\") pod \"55ceecfd-22bb-4683-94da-fb0077d5a22f\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") "
Sep 16 19:13:18.049719 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.049680    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ceecfd-22bb-4683-94da-fb0077d5a22f-tls-certs\") pod \"55ceecfd-22bb-4683-94da-fb0077d5a22f\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") "
Sep 16 19:13:18.049719 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.049713    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt6hf\" (UniqueName: \"kubernetes.io/projected/55ceecfd-22bb-4683-94da-fb0077d5a22f-kube-api-access-lt6hf\") pod \"55ceecfd-22bb-4683-94da-fb0077d5a22f\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") "
Sep 16 19:13:18.049926 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.049740    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-tmp-dir\") pod \"55ceecfd-22bb-4683-94da-fb0077d5a22f\" (UID: \"55ceecfd-22bb-4683-94da-fb0077d5a22f\") "
Sep 16 19:13:18.049987 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.049941    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-home" (OuterVolumeSpecName: "home") pod "55ceecfd-22bb-4683-94da-fb0077d5a22f" (UID: "55ceecfd-22bb-4683-94da-fb0077d5a22f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:18.049987 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.049952    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-model-cache" (OuterVolumeSpecName: "model-cache") pod "55ceecfd-22bb-4683-94da-fb0077d5a22f" (UID: "55ceecfd-22bb-4683-94da-fb0077d5a22f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:18.050096 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.050047    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55ceecfd-22bb-4683-94da-fb0077d5a22f" (UID: "55ceecfd-22bb-4683-94da-fb0077d5a22f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:18.050096 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.050065    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:13:18.050096 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.050083    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:13:18.051716 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.051687    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-dshm" (OuterVolumeSpecName: "dshm") pod "55ceecfd-22bb-4683-94da-fb0077d5a22f" (UID: "55ceecfd-22bb-4683-94da-fb0077d5a22f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:18.051716 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.051699    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ceecfd-22bb-4683-94da-fb0077d5a22f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55ceecfd-22bb-4683-94da-fb0077d5a22f" (UID: "55ceecfd-22bb-4683-94da-fb0077d5a22f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:18.051876 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.051726    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ceecfd-22bb-4683-94da-fb0077d5a22f-kube-api-access-lt6hf" (OuterVolumeSpecName: "kube-api-access-lt6hf") pod "55ceecfd-22bb-4683-94da-fb0077d5a22f" (UID: "55ceecfd-22bb-4683-94da-fb0077d5a22f"). InnerVolumeSpecName "kube-api-access-lt6hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:18.151094 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.151067    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ceecfd-22bb-4683-94da-fb0077d5a22f-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:13:18.151202 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.151097    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt6hf\" (UniqueName: \"kubernetes.io/projected/55ceecfd-22bb-4683-94da-fb0077d5a22f-kube-api-access-lt6hf\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:13:18.151202 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.151112    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:13:18.151202 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.151125    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ceecfd-22bb-4683-94da-fb0077d5a22f-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:13:18.207273 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.207214    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="621ed769-6e64-45a6-878f-5f99fff5e147" path="/var/lib/kubelet/pods/621ed769-6e64-45a6-878f-5f99fff5e147/volumes"
Sep 16 19:13:18.228271 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.228247    2565 generic.go:356] "Generic (PLEG): container finished" podID="55ceecfd-22bb-4683-94da-fb0077d5a22f" containerID="cd0c1e771f5d5c958ececee3966bbf73f7914143e8c21b44288f40618d9c92c6" exitCode=0
Sep 16 19:13:18.228383 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.228279    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"
Sep 16 19:13:18.228383 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.228305    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w" event={"ID":"55ceecfd-22bb-4683-94da-fb0077d5a22f","Type":"ContainerDied","Data":"cd0c1e771f5d5c958ececee3966bbf73f7914143e8c21b44288f40618d9c92c6"}
Sep 16 19:13:18.228383 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.228370    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w" event={"ID":"55ceecfd-22bb-4683-94da-fb0077d5a22f","Type":"ContainerDied","Data":"b94f3b75ab3b91e1cd3866fe026a28374a0a594dbe5f9cda4bde90f8b51093d0"}
Sep 16 19:13:18.228499 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.228393    2565 scope.go:117] "RemoveContainer" containerID="cd0c1e771f5d5c958ececee3966bbf73f7914143e8c21b44288f40618d9c92c6"
Sep 16 19:13:18.237065 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.237049    2565 scope.go:117] "RemoveContainer" containerID="cd0c1e771f5d5c958ececee3966bbf73f7914143e8c21b44288f40618d9c92c6"
Sep 16 19:13:18.237323 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:13:18.237298    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0c1e771f5d5c958ececee3966bbf73f7914143e8c21b44288f40618d9c92c6\": container with ID starting with cd0c1e771f5d5c958ececee3966bbf73f7914143e8c21b44288f40618d9c92c6 not found: ID does not exist" containerID="cd0c1e771f5d5c958ececee3966bbf73f7914143e8c21b44288f40618d9c92c6"
Sep 16 19:13:18.237391 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.237331    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0c1e771f5d5c958ececee3966bbf73f7914143e8c21b44288f40618d9c92c6"} err="failed to get container status \"cd0c1e771f5d5c958ececee3966bbf73f7914143e8c21b44288f40618d9c92c6\": rpc error: code = NotFound desc = could not find container \"cd0c1e771f5d5c958ececee3966bbf73f7914143e8c21b44288f40618d9c92c6\": container with ID starting with cd0c1e771f5d5c958ececee3966bbf73f7914143e8c21b44288f40618d9c92c6 not found: ID does not exist"
Sep 16 19:13:18.246495 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.246476    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"]
Sep 16 19:13:18.248900 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.248877    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-bc86w"]
Sep 16 19:13:18.764241 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.764218    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:18.776788 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:18.776767    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:19.252105 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:19.252081    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:13:19.991202 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:19.991171    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"]
Sep 16 19:13:19.991588 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:19.991573    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55ceecfd-22bb-4683-94da-fb0077d5a22f" containerName="main"
Sep 16 19:13:19.991588 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:19.991589    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="55ceecfd-22bb-4683-94da-fb0077d5a22f" containerName="main"
Sep 16 19:13:19.991690 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:19.991629    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="621ed769-6e64-45a6-878f-5f99fff5e147" containerName="main"
Sep 16 19:13:19.991690 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:19.991635    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="621ed769-6e64-45a6-878f-5f99fff5e147" containerName="main"
Sep 16 19:13:19.991766 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:19.991689    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="621ed769-6e64-45a6-878f-5f99fff5e147" containerName="main"
Sep 16 19:13:19.991766 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:19.991698    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="55ceecfd-22bb-4683-94da-fb0077d5a22f" containerName="main"
Sep 16 19:13:19.996830 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:19.996807    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:19.999255 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:19.999234    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Sep 16 19:13:20.010157 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.010137    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"]
Sep 16 19:13:20.166078 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.166048    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-model-cache\") pod \"tri-v2-kserve-dcb554bf5-9tpth\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:20.166211 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.166080    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-9tpth\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:20.166211 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.166100    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-9tpth\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:20.166295 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.166200    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-dshm\") pod \"tri-v2-kserve-dcb554bf5-9tpth\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:20.166295 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.166231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-home\") pod \"tri-v2-kserve-dcb554bf5-9tpth\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:20.166295 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.166258    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h2pn\" (UniqueName: \"kubernetes.io/projected/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-kube-api-access-4h2pn\") pod \"tri-v2-kserve-dcb554bf5-9tpth\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:20.207363 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.207322    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55ceecfd-22bb-4683-94da-fb0077d5a22f" path="/var/lib/kubelet/pods/55ceecfd-22bb-4683-94da-fb0077d5a22f/volumes"
Sep 16 19:13:20.267591 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.267523    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-dshm\") pod \"tri-v2-kserve-dcb554bf5-9tpth\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:20.267591 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.267558    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-home\") pod \"tri-v2-kserve-dcb554bf5-9tpth\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:20.267591 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.267587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h2pn\" (UniqueName: \"kubernetes.io/projected/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-kube-api-access-4h2pn\") pod \"tri-v2-kserve-dcb554bf5-9tpth\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:20.268095 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.267647    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-model-cache\") pod \"tri-v2-kserve-dcb554bf5-9tpth\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:20.268095 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.267678    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-9tpth\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:20.268095 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.267701    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-9tpth\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:20.268095 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.268035    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-home\") pod \"tri-v2-kserve-dcb554bf5-9tpth\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:20.268095 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.268068    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-model-cache\") pod \"tri-v2-kserve-dcb554bf5-9tpth\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:20.268266 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.268125    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-9tpth\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:20.269878 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.269860    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-dshm\") pod \"tri-v2-kserve-dcb554bf5-9tpth\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:20.270141 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.270124    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-9tpth\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:20.277428 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.277406    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h2pn\" (UniqueName: \"kubernetes.io/projected/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-kube-api-access-4h2pn\") pod \"tri-v2-kserve-dcb554bf5-9tpth\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:20.308911 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.308889    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:20.594088 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:20.594015    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"]
Sep 16 19:13:20.603459 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:13:20.603434    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7e9cfd_33eb_4cf2_897b_2dbb320140a5.slice/crio-5b7266fb7e9926fd932a1c309a388b097370463b94fe0358be752ae921476ccf WatchSource:0}: Error finding container 5b7266fb7e9926fd932a1c309a388b097370463b94fe0358be752ae921476ccf: Status 404 returned error can't find the container with id 5b7266fb7e9926fd932a1c309a388b097370463b94fe0358be752ae921476ccf
Sep 16 19:13:21.252060 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:21.252015    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth" event={"ID":"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5","Type":"ContainerStarted","Data":"b3b5b12c613496474036b4af0e29fff706f443fa89781d760b98da84eeb28a9c"}
Sep 16 19:13:21.252227 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:21.252078    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth" event={"ID":"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5","Type":"ContainerStarted","Data":"5b7266fb7e9926fd932a1c309a388b097370463b94fe0358be752ae921476ccf"}
Sep 16 19:13:21.271946 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:21.271897    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth" podStartSLOduration=2.271882668 podStartE2EDuration="2.271882668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:21.27015543 +0000 UTC m=+2039.629673426" watchObservedRunningTime="2026-09-16 19:13:21.271882668 +0000 UTC m=+2039.631400710"
Sep 16 19:13:21.309887 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:21.309859    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:30.309989 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:30.309958    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:30.322307 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:30.322282    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:31.315334 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:31.315302    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:13:36.361216 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.361183    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"]
Sep 16 19:13:36.364888 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.364858    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.373357 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.373320    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Sep 16 19:13:36.385244 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.385220    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"]
Sep 16 19:13:36.393264 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.393246    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-d9tvg\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.393392 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.393278    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-dshm\") pod \"tri-v3-kserve-798c88c66c-d9tvg\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.393392 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.393300    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-home\") pod \"tri-v3-kserve-798c88c66c-d9tvg\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.393392 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.393314    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-model-cache\") pod \"tri-v3-kserve-798c88c66c-d9tvg\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.393520 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.393407    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52364623-479d-4569-95ae-2b61a48e004b-tls-certs\") pod \"tri-v3-kserve-798c88c66c-d9tvg\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.393520 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.393437    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxgjn\" (UniqueName: \"kubernetes.io/projected/52364623-479d-4569-95ae-2b61a48e004b-kube-api-access-nxgjn\") pod \"tri-v3-kserve-798c88c66c-d9tvg\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.494393 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.494370    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-d9tvg\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.494501 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.494410    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-dshm\") pod \"tri-v3-kserve-798c88c66c-d9tvg\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.494501 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.494449    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-home\") pod \"tri-v3-kserve-798c88c66c-d9tvg\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.494501 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.494464    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-model-cache\") pod \"tri-v3-kserve-798c88c66c-d9tvg\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.494501 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.494494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52364623-479d-4569-95ae-2b61a48e004b-tls-certs\") pod \"tri-v3-kserve-798c88c66c-d9tvg\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.494705 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.494524    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxgjn\" (UniqueName: \"kubernetes.io/projected/52364623-479d-4569-95ae-2b61a48e004b-kube-api-access-nxgjn\") pod \"tri-v3-kserve-798c88c66c-d9tvg\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.494810 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.494787    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-d9tvg\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.494870 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.494812    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-model-cache\") pod \"tri-v3-kserve-798c88c66c-d9tvg\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.494932 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.494875    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-home\") pod \"tri-v3-kserve-798c88c66c-d9tvg\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.496981 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.496958    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-dshm\") pod \"tri-v3-kserve-798c88c66c-d9tvg\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.497112 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.497035    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52364623-479d-4569-95ae-2b61a48e004b-tls-certs\") pod \"tri-v3-kserve-798c88c66c-d9tvg\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.506403 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.506381    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxgjn\" (UniqueName: \"kubernetes.io/projected/52364623-479d-4569-95ae-2b61a48e004b-kube-api-access-nxgjn\") pod \"tri-v3-kserve-798c88c66c-d9tvg\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.675295 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.675207    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:36.955307 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:36.955277    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"]
Sep 16 19:13:36.956871 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:13:36.956848    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52364623_479d_4569_95ae_2b61a48e004b.slice/crio-a1c8c5f5c35017f6a875473dcb66ec8ce934b4eae6d9aec86aefbf995297ecfc WatchSource:0}: Error finding container a1c8c5f5c35017f6a875473dcb66ec8ce934b4eae6d9aec86aefbf995297ecfc: Status 404 returned error can't find the container with id a1c8c5f5c35017f6a875473dcb66ec8ce934b4eae6d9aec86aefbf995297ecfc
Sep 16 19:13:37.335809 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:37.335774    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg" event={"ID":"52364623-479d-4569-95ae-2b61a48e004b","Type":"ContainerStarted","Data":"ad59611703f3a7c1977a02b440974d2345a00ba21be726477885b6013bb14a9b"}
Sep 16 19:13:37.335976 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:37.335826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg" event={"ID":"52364623-479d-4569-95ae-2b61a48e004b","Type":"ContainerStarted","Data":"a1c8c5f5c35017f6a875473dcb66ec8ce934b4eae6d9aec86aefbf995297ecfc"}
Sep 16 19:13:37.361280 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:37.361212    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg" podStartSLOduration=1.3611926429999999 podStartE2EDuration="1.361192643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:37.357247096 +0000 UTC m=+2055.716765092" watchObservedRunningTime="2026-09-16 19:13:37.361192643 +0000 UTC m=+2055.720710642"
Sep 16 19:13:37.675926 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:37.675852    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:46.675431 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:46.675334    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:46.689433 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:46.689411    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:13:47.401294 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:13:47.401260    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:15:02.612027 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:02.611989    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"]
Sep 16 19:15:02.616646 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:02.616617    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"]
Sep 16 19:15:02.625147 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:02.625118    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"]
Sep 16 19:15:17.631932 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.631820    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq" podUID="c2612890-d754-4c96-b63e-e5353b41dc0c" containerName="main" containerID="cri-o://fc475342cd23f877c66f1ae85303a3d040ad480c83cc9b9b1092964835d5812d" gracePeriod=45
Sep 16 19:15:17.635876 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.635843    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth" podUID="2e7e9cfd-33eb-4cf2-897b-2dbb320140a5" containerName="main" containerID="cri-o://b3b5b12c613496474036b4af0e29fff706f443fa89781d760b98da84eeb28a9c" gracePeriod=45
Sep 16 19:15:17.825086 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.825025    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg" podUID="52364623-479d-4569-95ae-2b61a48e004b" containerName="main" containerID="cri-o://ad59611703f3a7c1977a02b440974d2345a00ba21be726477885b6013bb14a9b" gracePeriod=45
Sep 16 19:15:17.893992 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.893952    2565 generic.go:356] "Generic (PLEG): container finished" podID="2e7e9cfd-33eb-4cf2-897b-2dbb320140a5" containerID="b3b5b12c613496474036b4af0e29fff706f443fa89781d760b98da84eeb28a9c" exitCode=0
Sep 16 19:15:17.894121 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.894027    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth" event={"ID":"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5","Type":"ContainerDied","Data":"b3b5b12c613496474036b4af0e29fff706f443fa89781d760b98da84eeb28a9c"}
Sep 16 19:15:17.895749 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.895729    2565 generic.go:356] "Generic (PLEG): container finished" podID="c2612890-d754-4c96-b63e-e5353b41dc0c" containerID="fc475342cd23f877c66f1ae85303a3d040ad480c83cc9b9b1092964835d5812d" exitCode=0
Sep 16 19:15:17.895851 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.895758    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq" event={"ID":"c2612890-d754-4c96-b63e-e5353b41dc0c","Type":"ContainerDied","Data":"fc475342cd23f877c66f1ae85303a3d040ad480c83cc9b9b1092964835d5812d"}
Sep 16 19:15:17.937802 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.937783    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:15:17.941152 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.941135    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:15:17.977888 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.977850    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-tmp-dir\") pod \"c2612890-d754-4c96-b63e-e5353b41dc0c\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") "
Sep 16 19:15:17.978050 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.977897    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-dshm\") pod \"c2612890-d754-4c96-b63e-e5353b41dc0c\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") "
Sep 16 19:15:17.978050 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.977913    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-model-cache\") pod \"c2612890-d754-4c96-b63e-e5353b41dc0c\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") "
Sep 16 19:15:17.978050 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.977935    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h2pn\" (UniqueName: \"kubernetes.io/projected/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-kube-api-access-4h2pn\") pod \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") "
Sep 16 19:15:17.978050 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.977962    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-tmp-dir\") pod \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") "
Sep 16 19:15:17.978050 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.978022    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-dshm\") pod \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") "
Sep 16 19:15:17.978340 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.978057    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-tls-certs\") pod \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") "
Sep 16 19:15:17.978340 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.978089    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-home\") pod \"c2612890-d754-4c96-b63e-e5353b41dc0c\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") "
Sep 16 19:15:17.978340 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.978130    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-model-cache\") pod \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") "
Sep 16 19:15:17.978340 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.978168    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2612890-d754-4c96-b63e-e5353b41dc0c-tls-certs\") pod \"c2612890-d754-4c96-b63e-e5353b41dc0c\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") "
Sep 16 19:15:17.978340 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.978199    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h52dk\" (UniqueName: \"kubernetes.io/projected/c2612890-d754-4c96-b63e-e5353b41dc0c-kube-api-access-h52dk\") pod \"c2612890-d754-4c96-b63e-e5353b41dc0c\" (UID: \"c2612890-d754-4c96-b63e-e5353b41dc0c\") "
Sep 16 19:15:17.978340 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.978222    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-home\") pod \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\" (UID: \"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5\") "
Sep 16 19:15:17.978340 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.978264    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e7e9cfd-33eb-4cf2-897b-2dbb320140a5" (UID: "2e7e9cfd-33eb-4cf2-897b-2dbb320140a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:17.978723 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.978453    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e7e9cfd-33eb-4cf2-897b-2dbb320140a5" (UID: "2e7e9cfd-33eb-4cf2-897b-2dbb320140a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:17.978723 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.978567    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:15:17.978723 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.978587    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:15:17.978723 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.978639    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-home" (OuterVolumeSpecName: "home") pod "2e7e9cfd-33eb-4cf2-897b-2dbb320140a5" (UID: "2e7e9cfd-33eb-4cf2-897b-2dbb320140a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:17.979955 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.979879    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-home" (OuterVolumeSpecName: "home") pod "c2612890-d754-4c96-b63e-e5353b41dc0c" (UID: "c2612890-d754-4c96-b63e-e5353b41dc0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:17.980141 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.979988    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "c2612890-d754-4c96-b63e-e5353b41dc0c" (UID: "c2612890-d754-4c96-b63e-e5353b41dc0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:17.980278 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.980238    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c2612890-d754-4c96-b63e-e5353b41dc0c" (UID: "c2612890-d754-4c96-b63e-e5353b41dc0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:17.980944 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.980745    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-dshm" (OuterVolumeSpecName: "dshm") pod "2e7e9cfd-33eb-4cf2-897b-2dbb320140a5" (UID: "2e7e9cfd-33eb-4cf2-897b-2dbb320140a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:17.980944 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.980838    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e7e9cfd-33eb-4cf2-897b-2dbb320140a5" (UID: "2e7e9cfd-33eb-4cf2-897b-2dbb320140a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:17.981816 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.981784    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-kube-api-access-4h2pn" (OuterVolumeSpecName: "kube-api-access-4h2pn") pod "2e7e9cfd-33eb-4cf2-897b-2dbb320140a5" (UID: "2e7e9cfd-33eb-4cf2-897b-2dbb320140a5"). InnerVolumeSpecName "kube-api-access-4h2pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:17.982047 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.982008    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2612890-d754-4c96-b63e-e5353b41dc0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c2612890-d754-4c96-b63e-e5353b41dc0c" (UID: "c2612890-d754-4c96-b63e-e5353b41dc0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:17.982133 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.982062    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-dshm" (OuterVolumeSpecName: "dshm") pod "c2612890-d754-4c96-b63e-e5353b41dc0c" (UID: "c2612890-d754-4c96-b63e-e5353b41dc0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:17.982262 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:17.982239    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2612890-d754-4c96-b63e-e5353b41dc0c-kube-api-access-h52dk" (OuterVolumeSpecName: "kube-api-access-h52dk") pod "c2612890-d754-4c96-b63e-e5353b41dc0c" (UID: "c2612890-d754-4c96-b63e-e5353b41dc0c"). InnerVolumeSpecName "kube-api-access-h52dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:18.079508 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.079483    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2612890-d754-4c96-b63e-e5353b41dc0c-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:15:18.079627 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.079515    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h52dk\" (UniqueName: \"kubernetes.io/projected/c2612890-d754-4c96-b63e-e5353b41dc0c-kube-api-access-h52dk\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:15:18.079627 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.079530    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:15:18.079627 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.079543    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:15:18.079627 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.079556    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:15:18.079627 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.079567    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:15:18.079627 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.079578    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h2pn\" (UniqueName: \"kubernetes.io/projected/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-kube-api-access-4h2pn\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:15:18.079627 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.079589    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:15:18.079627 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.079601    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:15:18.079627 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.079611    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2612890-d754-4c96-b63e-e5353b41dc0c-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:15:18.084225 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.084203    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:15:18.180569 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.180502    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52364623-479d-4569-95ae-2b61a48e004b-tls-certs\") pod \"52364623-479d-4569-95ae-2b61a48e004b\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") "
Sep 16 19:15:18.180569 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.180558    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-dshm\") pod \"52364623-479d-4569-95ae-2b61a48e004b\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") "
Sep 16 19:15:18.180728 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.180600    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxgjn\" (UniqueName: \"kubernetes.io/projected/52364623-479d-4569-95ae-2b61a48e004b-kube-api-access-nxgjn\") pod \"52364623-479d-4569-95ae-2b61a48e004b\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") "
Sep 16 19:15:18.180728 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.180628    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-tmp-dir\") pod \"52364623-479d-4569-95ae-2b61a48e004b\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") "
Sep 16 19:15:18.180728 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.180647    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-home\") pod \"52364623-479d-4569-95ae-2b61a48e004b\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") "
Sep 16 19:15:18.180728 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.180672    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-model-cache\") pod \"52364623-479d-4569-95ae-2b61a48e004b\" (UID: \"52364623-479d-4569-95ae-2b61a48e004b\") "
Sep 16 19:15:18.180985 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.180923    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52364623-479d-4569-95ae-2b61a48e004b" (UID: "52364623-479d-4569-95ae-2b61a48e004b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:18.181057 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.180936    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-home" (OuterVolumeSpecName: "home") pod "52364623-479d-4569-95ae-2b61a48e004b" (UID: "52364623-479d-4569-95ae-2b61a48e004b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:18.181057 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.180990    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-model-cache" (OuterVolumeSpecName: "model-cache") pod "52364623-479d-4569-95ae-2b61a48e004b" (UID: "52364623-479d-4569-95ae-2b61a48e004b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:18.182684 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.182656    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-dshm" (OuterVolumeSpecName: "dshm") pod "52364623-479d-4569-95ae-2b61a48e004b" (UID: "52364623-479d-4569-95ae-2b61a48e004b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:18.182770 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.182699    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52364623-479d-4569-95ae-2b61a48e004b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52364623-479d-4569-95ae-2b61a48e004b" (UID: "52364623-479d-4569-95ae-2b61a48e004b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:18.182770 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.182702    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52364623-479d-4569-95ae-2b61a48e004b-kube-api-access-nxgjn" (OuterVolumeSpecName: "kube-api-access-nxgjn") pod "52364623-479d-4569-95ae-2b61a48e004b" (UID: "52364623-479d-4569-95ae-2b61a48e004b"). InnerVolumeSpecName "kube-api-access-nxgjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:18.281361 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.281327    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:15:18.281462 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.281363    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxgjn\" (UniqueName: \"kubernetes.io/projected/52364623-479d-4569-95ae-2b61a48e004b-kube-api-access-nxgjn\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:15:18.281462 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.281374    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:15:18.281462 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.281383    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:15:18.281462 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.281390    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52364623-479d-4569-95ae-2b61a48e004b-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:15:18.281462 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.281400    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52364623-479d-4569-95ae-2b61a48e004b-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:15:18.902437 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.902382    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"
Sep 16 19:15:18.902825 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.902382    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth" event={"ID":"2e7e9cfd-33eb-4cf2-897b-2dbb320140a5","Type":"ContainerDied","Data":"5b7266fb7e9926fd932a1c309a388b097370463b94fe0358be752ae921476ccf"}
Sep 16 19:15:18.902825 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.902525    2565 scope.go:117] "RemoveContainer" containerID="b3b5b12c613496474036b4af0e29fff706f443fa89781d760b98da84eeb28a9c"
Sep 16 19:15:18.903923 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.903826    2565 generic.go:356] "Generic (PLEG): container finished" podID="52364623-479d-4569-95ae-2b61a48e004b" containerID="ad59611703f3a7c1977a02b440974d2345a00ba21be726477885b6013bb14a9b" exitCode=0
Sep 16 19:15:18.903923 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.903854    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg" event={"ID":"52364623-479d-4569-95ae-2b61a48e004b","Type":"ContainerDied","Data":"ad59611703f3a7c1977a02b440974d2345a00ba21be726477885b6013bb14a9b"}
Sep 16 19:15:18.903923 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.903872    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"
Sep 16 19:15:18.903923 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.903906    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg" event={"ID":"52364623-479d-4569-95ae-2b61a48e004b","Type":"ContainerDied","Data":"a1c8c5f5c35017f6a875473dcb66ec8ce934b4eae6d9aec86aefbf995297ecfc"}
Sep 16 19:15:18.905320 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.905296    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq" event={"ID":"c2612890-d754-4c96-b63e-e5353b41dc0c","Type":"ContainerDied","Data":"cfe5ac2ccd1d7196f4945b6550cee9e451909d52cbfb8426028e372d08790c87"}
Sep 16 19:15:18.905445 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.905326    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"
Sep 16 19:15:18.912053 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.912036    2565 scope.go:117] "RemoveContainer" containerID="ad59611703f3a7c1977a02b440974d2345a00ba21be726477885b6013bb14a9b"
Sep 16 19:15:18.919608 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.919583    2565 scope.go:117] "RemoveContainer" containerID="ad59611703f3a7c1977a02b440974d2345a00ba21be726477885b6013bb14a9b"
Sep 16 19:15:18.919871 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:15:18.919850    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad59611703f3a7c1977a02b440974d2345a00ba21be726477885b6013bb14a9b\": container with ID starting with ad59611703f3a7c1977a02b440974d2345a00ba21be726477885b6013bb14a9b not found: ID does not exist" containerID="ad59611703f3a7c1977a02b440974d2345a00ba21be726477885b6013bb14a9b"
Sep 16 19:15:18.919939 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.919878    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad59611703f3a7c1977a02b440974d2345a00ba21be726477885b6013bb14a9b"} err="failed to get container status \"ad59611703f3a7c1977a02b440974d2345a00ba21be726477885b6013bb14a9b\": rpc error: code = NotFound desc = could not find container \"ad59611703f3a7c1977a02b440974d2345a00ba21be726477885b6013bb14a9b\": container with ID starting with ad59611703f3a7c1977a02b440974d2345a00ba21be726477885b6013bb14a9b not found: ID does not exist"
Sep 16 19:15:18.919939 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.919894    2565 scope.go:117] "RemoveContainer" containerID="fc475342cd23f877c66f1ae85303a3d040ad480c83cc9b9b1092964835d5812d"
Sep 16 19:15:18.933531 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.933512    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"]
Sep 16 19:15:18.937886 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.937867    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-4zhhq"]
Sep 16 19:15:18.948185 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.948165    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"]
Sep 16 19:15:18.954164 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.954142    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9tpth"]
Sep 16 19:15:18.968767 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.968739    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"]
Sep 16 19:15:18.978670 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:18.978650    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-d9tvg"]
Sep 16 19:15:20.207303 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:20.207272    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e7e9cfd-33eb-4cf2-897b-2dbb320140a5" path="/var/lib/kubelet/pods/2e7e9cfd-33eb-4cf2-897b-2dbb320140a5/volumes"
Sep 16 19:15:20.207693 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:20.207678    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52364623-479d-4569-95ae-2b61a48e004b" path="/var/lib/kubelet/pods/52364623-479d-4569-95ae-2b61a48e004b/volumes"
Sep 16 19:15:20.208045 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:20.208034    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2612890-d754-4c96-b63e-e5353b41dc0c" path="/var/lib/kubelet/pods/c2612890-d754-4c96-b63e-e5353b41dc0c/volumes"
Sep 16 19:15:22.056603 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.056567    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"]
Sep 16 19:15:22.057011 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.056997    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2612890-d754-4c96-b63e-e5353b41dc0c" containerName="main"
Sep 16 19:15:22.057055 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.057013    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2612890-d754-4c96-b63e-e5353b41dc0c" containerName="main"
Sep 16 19:15:22.057055 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.057022    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e7e9cfd-33eb-4cf2-897b-2dbb320140a5" containerName="main"
Sep 16 19:15:22.057055 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.057028    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e7e9cfd-33eb-4cf2-897b-2dbb320140a5" containerName="main"
Sep 16 19:15:22.057055 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.057037    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52364623-479d-4569-95ae-2b61a48e004b" containerName="main"
Sep 16 19:15:22.057055 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.057046    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="52364623-479d-4569-95ae-2b61a48e004b" containerName="main"
Sep 16 19:15:22.057211 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.057113    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="52364623-479d-4569-95ae-2b61a48e004b" containerName="main"
Sep 16 19:15:22.057211 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.057125    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2612890-d754-4c96-b63e-e5353b41dc0c" containerName="main"
Sep 16 19:15:22.057211 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.057132    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e7e9cfd-33eb-4cf2-897b-2dbb320140a5" containerName="main"
Sep 16 19:15:22.061820 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.061797    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.065164 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.065143    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 16 19:15:22.065313 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.065147    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 16 19:15:22.066052 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.066029    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-bwc2w\""
Sep 16 19:15:22.066137 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.066072    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Sep 16 19:15:22.074245 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.074223    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"]
Sep 16 19:15:22.108752 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.108722    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-tls-certs\") pod \"late-v1-kserve-56cbc8456c-l64kh\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.108885 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.108774    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-model-cache\") pod \"late-v1-kserve-56cbc8456c-l64kh\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.108933 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.108886    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-dshm\") pod \"late-v1-kserve-56cbc8456c-l64kh\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.108985 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.108941    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-home\") pod \"late-v1-kserve-56cbc8456c-l64kh\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.109034 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.108991    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-l64kh\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.109034 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.109022    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2pw5\" (UniqueName: \"kubernetes.io/projected/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-kube-api-access-r2pw5\") pod \"late-v1-kserve-56cbc8456c-l64kh\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.209825 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.209800    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-dshm\") pod \"late-v1-kserve-56cbc8456c-l64kh\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.209970 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.209837    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-home\") pod \"late-v1-kserve-56cbc8456c-l64kh\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.209970 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.209859    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-l64kh\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.209970 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.209882    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2pw5\" (UniqueName: \"kubernetes.io/projected/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-kube-api-access-r2pw5\") pod \"late-v1-kserve-56cbc8456c-l64kh\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.209970 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.209921    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-tls-certs\") pod \"late-v1-kserve-56cbc8456c-l64kh\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.209970 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.209938    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-model-cache\") pod \"late-v1-kserve-56cbc8456c-l64kh\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.210218 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.210178    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-home\") pod \"late-v1-kserve-56cbc8456c-l64kh\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.210303 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.210276    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-l64kh\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.210390 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.210370    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-model-cache\") pod \"late-v1-kserve-56cbc8456c-l64kh\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.212103 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.212081    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-dshm\") pod \"late-v1-kserve-56cbc8456c-l64kh\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.212231 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.212214    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-tls-certs\") pod \"late-v1-kserve-56cbc8456c-l64kh\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.222180 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.222152    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2pw5\" (UniqueName: \"kubernetes.io/projected/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-kube-api-access-r2pw5\") pod \"late-v1-kserve-56cbc8456c-l64kh\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.373504 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.373453    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:22.654676 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.654648    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"]
Sep 16 19:15:22.655425 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:15:22.655398    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c11510_c2bf_4d97_8ee9_56aec8460fc3.slice/crio-b6b6ec9d6506b8cfccd9c03b4d4a933bfc5452a098f22b8cfe6f5e4565fdbe45 WatchSource:0}: Error finding container b6b6ec9d6506b8cfccd9c03b4d4a933bfc5452a098f22b8cfe6f5e4565fdbe45: Status 404 returned error can't find the container with id b6b6ec9d6506b8cfccd9c03b4d4a933bfc5452a098f22b8cfe6f5e4565fdbe45
Sep 16 19:15:22.931704 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.931613    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh" event={"ID":"c3c11510-c2bf-4d97-8ee9-56aec8460fc3","Type":"ContainerStarted","Data":"d30893c75ac50fce6ae93bb9867320aaa85c8957d43f5fd6c985876a322eedba"}
Sep 16 19:15:22.931704 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.931682    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh" event={"ID":"c3c11510-c2bf-4d97-8ee9-56aec8460fc3","Type":"ContainerStarted","Data":"b6b6ec9d6506b8cfccd9c03b4d4a933bfc5452a098f22b8cfe6f5e4565fdbe45"}
Sep 16 19:15:22.952554 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:22.952503    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh" podStartSLOduration=0.952488276 podStartE2EDuration="952.488276ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:22.950224939 +0000 UTC m=+2161.309742926" watchObservedRunningTime="2026-09-16 19:15:22.952488276 +0000 UTC m=+2161.312006271"
Sep 16 19:15:23.374274 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:23.374244    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:32.374318 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:32.374286    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:32.386430 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:32.386403    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:15:32.995323 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:15:32.995294    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:16:38.834927 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:38.834892    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"]
Sep 16 19:16:50.457392 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.457256    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"]
Sep 16 19:16:50.461165 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.461144    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:50.463816 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.463787    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 16 19:16:50.463951 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.463907    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 16 19:16:50.464955 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.464935    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Sep 16 19:16:50.465069 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.465002    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-vxgl4\""
Sep 16 19:16:50.477056 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.474709    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"]
Sep 16 19:16:50.480966 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.480943    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-qhdvl\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:50.481075 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.481001    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-dshm\") pod \"delw-v1-kserve-6cf8cbb847-qhdvl\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:50.481075 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.481028    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c848259e-7f7b-4b41-b43b-ad537df2a11a-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-qhdvl\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:50.481075 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.481071    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-qhdvl\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:50.481202 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.481098    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-home\") pod \"delw-v1-kserve-6cf8cbb847-qhdvl\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:50.481202 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.481157    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n5cs\" (UniqueName: \"kubernetes.io/projected/c848259e-7f7b-4b41-b43b-ad537df2a11a-kube-api-access-2n5cs\") pod \"delw-v1-kserve-6cf8cbb847-qhdvl\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:50.582297 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.582268    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-qhdvl\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:50.582477 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.582312    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-dshm\") pod \"delw-v1-kserve-6cf8cbb847-qhdvl\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:50.582477 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.582340    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c848259e-7f7b-4b41-b43b-ad537df2a11a-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-qhdvl\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:50.582477 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.582417    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-qhdvl\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:50.582477 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.582453    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-home\") pod \"delw-v1-kserve-6cf8cbb847-qhdvl\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:50.582702 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.582504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n5cs\" (UniqueName: \"kubernetes.io/projected/c848259e-7f7b-4b41-b43b-ad537df2a11a-kube-api-access-2n5cs\") pod \"delw-v1-kserve-6cf8cbb847-qhdvl\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:50.582760 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.582731    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-qhdvl\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:50.582845 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.582824    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-home\") pod \"delw-v1-kserve-6cf8cbb847-qhdvl\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:50.582900 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.582881    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-qhdvl\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:50.585065 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.585039    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-dshm\") pod \"delw-v1-kserve-6cf8cbb847-qhdvl\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:50.585277 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.585258    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c848259e-7f7b-4b41-b43b-ad537df2a11a-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-qhdvl\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:50.592854 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.592824    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n5cs\" (UniqueName: \"kubernetes.io/projected/c848259e-7f7b-4b41-b43b-ad537df2a11a-kube-api-access-2n5cs\") pod \"delw-v1-kserve-6cf8cbb847-qhdvl\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:50.778040 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:50.778007    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:51.073397 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:51.073375    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"]
Sep 16 19:16:51.075597 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:16:51.075567    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc848259e_7f7b_4b41_b43b_ad537df2a11a.slice/crio-c858aa28b9ab9eff82e5415de750b48e44011f200e8bfa2c6cd6ec777042ba2b WatchSource:0}: Error finding container c858aa28b9ab9eff82e5415de750b48e44011f200e8bfa2c6cd6ec777042ba2b: Status 404 returned error can't find the container with id c858aa28b9ab9eff82e5415de750b48e44011f200e8bfa2c6cd6ec777042ba2b
Sep 16 19:16:51.429516 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:51.429423    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl" event={"ID":"c848259e-7f7b-4b41-b43b-ad537df2a11a","Type":"ContainerStarted","Data":"48dd7372689f414af7514f51c3ae00dc6bcf1450872484ab47c07ad0499f123d"}
Sep 16 19:16:51.429516 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:51.429494    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl" event={"ID":"c848259e-7f7b-4b41-b43b-ad537df2a11a","Type":"ContainerStarted","Data":"c858aa28b9ab9eff82e5415de750b48e44011f200e8bfa2c6cd6ec777042ba2b"}
Sep 16 19:16:51.460560 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:51.460507    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl" podStartSLOduration=1.460491706 podStartE2EDuration="1.460491706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:51.457767776 +0000 UTC m=+2249.817285776" watchObservedRunningTime="2026-09-16 19:16:51.460491706 +0000 UTC m=+2249.820009700"
Sep 16 19:16:51.778824 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:51.778797    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:16:53.854362 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:53.854308    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh" podUID="c3c11510-c2bf-4d97-8ee9-56aec8460fc3" containerName="main" containerID="cri-o://d30893c75ac50fce6ae93bb9867320aaa85c8957d43f5fd6c985876a322eedba" gracePeriod=45
Sep 16 19:16:53.984655 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:53.984618    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh" podUID="c3c11510-c2bf-4d97-8ee9-56aec8460fc3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 16 19:16:54.114942 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.114889    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:16:54.213004 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.212980    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-home\") pod \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") "
Sep 16 19:16:54.213160 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.213025    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2pw5\" (UniqueName: \"kubernetes.io/projected/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-kube-api-access-r2pw5\") pod \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") "
Sep 16 19:16:54.213160 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.213051    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-tmp-dir\") pod \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") "
Sep 16 19:16:54.213160 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.213114    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-tls-certs\") pod \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") "
Sep 16 19:16:54.213160 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.213151    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-model-cache\") pod \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") "
Sep 16 19:16:54.213412 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.213170    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-dshm\") pod \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\" (UID: \"c3c11510-c2bf-4d97-8ee9-56aec8460fc3\") "
Sep 16 19:16:54.213412 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.213275    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-home" (OuterVolumeSpecName: "home") pod "c3c11510-c2bf-4d97-8ee9-56aec8460fc3" (UID: "c3c11510-c2bf-4d97-8ee9-56aec8460fc3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:54.213412 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.213333    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3c11510-c2bf-4d97-8ee9-56aec8460fc3" (UID: "c3c11510-c2bf-4d97-8ee9-56aec8460fc3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:54.213581 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.213424    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:16:54.213581 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.213456    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3c11510-c2bf-4d97-8ee9-56aec8460fc3" (UID: "c3c11510-c2bf-4d97-8ee9-56aec8460fc3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:54.215282 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.215261    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3c11510-c2bf-4d97-8ee9-56aec8460fc3" (UID: "c3c11510-c2bf-4d97-8ee9-56aec8460fc3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:54.215384 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.215287    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-dshm" (OuterVolumeSpecName: "dshm") pod "c3c11510-c2bf-4d97-8ee9-56aec8460fc3" (UID: "c3c11510-c2bf-4d97-8ee9-56aec8460fc3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:54.215440 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.215396    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-kube-api-access-r2pw5" (OuterVolumeSpecName: "kube-api-access-r2pw5") pod "c3c11510-c2bf-4d97-8ee9-56aec8460fc3" (UID: "c3c11510-c2bf-4d97-8ee9-56aec8460fc3"). InnerVolumeSpecName "kube-api-access-r2pw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:54.314024 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.313998    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:16:54.314024 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.314020    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:16:54.314159 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.314028    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:16:54.314159 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.314037    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2pw5\" (UniqueName: \"kubernetes.io/projected/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-kube-api-access-r2pw5\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:16:54.314159 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.314046    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3c11510-c2bf-4d97-8ee9-56aec8460fc3-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:16:54.447697 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.447624    2565 generic.go:356] "Generic (PLEG): container finished" podID="c3c11510-c2bf-4d97-8ee9-56aec8460fc3" containerID="d30893c75ac50fce6ae93bb9867320aaa85c8957d43f5fd6c985876a322eedba" exitCode=0
Sep 16 19:16:54.447697 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.447671    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"
Sep 16 19:16:54.447836 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.447713    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh" event={"ID":"c3c11510-c2bf-4d97-8ee9-56aec8460fc3","Type":"ContainerDied","Data":"d30893c75ac50fce6ae93bb9867320aaa85c8957d43f5fd6c985876a322eedba"}
Sep 16 19:16:54.447836 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.447767    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh" event={"ID":"c3c11510-c2bf-4d97-8ee9-56aec8460fc3","Type":"ContainerDied","Data":"b6b6ec9d6506b8cfccd9c03b4d4a933bfc5452a098f22b8cfe6f5e4565fdbe45"}
Sep 16 19:16:54.447836 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.447791    2565 scope.go:117] "RemoveContainer" containerID="d30893c75ac50fce6ae93bb9867320aaa85c8957d43f5fd6c985876a322eedba"
Sep 16 19:16:54.457087 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.457071    2565 scope.go:117] "RemoveContainer" containerID="d30893c75ac50fce6ae93bb9867320aaa85c8957d43f5fd6c985876a322eedba"
Sep 16 19:16:54.457394 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:16:54.457373    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30893c75ac50fce6ae93bb9867320aaa85c8957d43f5fd6c985876a322eedba\": container with ID starting with d30893c75ac50fce6ae93bb9867320aaa85c8957d43f5fd6c985876a322eedba not found: ID does not exist" containerID="d30893c75ac50fce6ae93bb9867320aaa85c8957d43f5fd6c985876a322eedba"
Sep 16 19:16:54.457464 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.457404    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30893c75ac50fce6ae93bb9867320aaa85c8957d43f5fd6c985876a322eedba"} err="failed to get container status \"d30893c75ac50fce6ae93bb9867320aaa85c8957d43f5fd6c985876a322eedba\": rpc error: code = NotFound desc = could not find container \"d30893c75ac50fce6ae93bb9867320aaa85c8957d43f5fd6c985876a322eedba\": container with ID starting with d30893c75ac50fce6ae93bb9867320aaa85c8957d43f5fd6c985876a322eedba not found: ID does not exist"
Sep 16 19:16:54.482630 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.482607    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"]
Sep 16 19:16:54.486121 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:54.486099    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-l64kh"]
Sep 16 19:16:56.207165 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:16:56.207132    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3c11510-c2bf-4d97-8ee9-56aec8460fc3" path="/var/lib/kubelet/pods/c3c11510-c2bf-4d97-8ee9-56aec8460fc3/volumes"
Sep 16 19:17:00.778394 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:00.778339    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:17:00.791039 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:00.791010    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:17:01.501032 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:01.500993    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:17:15.401044 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.401003    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"]
Sep 16 19:17:15.401457 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.401402    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3c11510-c2bf-4d97-8ee9-56aec8460fc3" containerName="main"
Sep 16 19:17:15.401457 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.401414    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3c11510-c2bf-4d97-8ee9-56aec8460fc3" containerName="main"
Sep 16 19:17:15.401540 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.401488    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3c11510-c2bf-4d97-8ee9-56aec8460fc3" containerName="main"
Sep 16 19:17:15.404633 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.404614    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.408472 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.408447    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Sep 16 19:17:15.417110 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.417085    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"]
Sep 16 19:17:15.473508 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.473481    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-ckfpv\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.473630 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.473520    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa415c9d-ec09-493d-b9ce-6712f79911e3-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-ckfpv\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.473630 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.473571    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-home\") pod \"delw-v2-kserve-5b84cc7ddf-ckfpv\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.473630 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.473595    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-ckfpv\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.473735 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.473706    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5nbl\" (UniqueName: \"kubernetes.io/projected/aa415c9d-ec09-493d-b9ce-6712f79911e3-kube-api-access-l5nbl\") pod \"delw-v2-kserve-5b84cc7ddf-ckfpv\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.473772 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.473753    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-ckfpv\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.574125 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.574099    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5nbl\" (UniqueName: \"kubernetes.io/projected/aa415c9d-ec09-493d-b9ce-6712f79911e3-kube-api-access-l5nbl\") pod \"delw-v2-kserve-5b84cc7ddf-ckfpv\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.574240 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.574141    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-ckfpv\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.574240 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.574172    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-ckfpv\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.574240 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.574208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa415c9d-ec09-493d-b9ce-6712f79911e3-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-ckfpv\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.574432 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.574259    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-home\") pod \"delw-v2-kserve-5b84cc7ddf-ckfpv\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.574432 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.574292    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-ckfpv\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.574663 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.574634    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-ckfpv\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.574767 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.574665    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-ckfpv\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.574767 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.574711    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-home\") pod \"delw-v2-kserve-5b84cc7ddf-ckfpv\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.576516 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.576488    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-ckfpv\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.576768 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.576745    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa415c9d-ec09-493d-b9ce-6712f79911e3-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-ckfpv\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.591325 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.591302    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5nbl\" (UniqueName: \"kubernetes.io/projected/aa415c9d-ec09-493d-b9ce-6712f79911e3-kube-api-access-l5nbl\") pod \"delw-v2-kserve-5b84cc7ddf-ckfpv\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.716366 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.716284    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:15.996489 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:15.996461    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"]
Sep 16 19:17:15.998486 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:17:15.998459    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa415c9d_ec09_493d_b9ce_6712f79911e3.slice/crio-df5bb6a2561f80d4efbeb11dbab97c38cf2b9c115b00a2ccc6752c3e2d552a3b WatchSource:0}: Error finding container df5bb6a2561f80d4efbeb11dbab97c38cf2b9c115b00a2ccc6752c3e2d552a3b: Status 404 returned error can't find the container with id df5bb6a2561f80d4efbeb11dbab97c38cf2b9c115b00a2ccc6752c3e2d552a3b
Sep 16 19:17:16.570212 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:16.570172    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv" event={"ID":"aa415c9d-ec09-493d-b9ce-6712f79911e3","Type":"ContainerStarted","Data":"78e015a32b4de008da272dfcf641241c18a4a3d2bbdd888de91808cac7720a0c"}
Sep 16 19:17:16.570674 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:16.570256    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv" event={"ID":"aa415c9d-ec09-493d-b9ce-6712f79911e3","Type":"ContainerStarted","Data":"df5bb6a2561f80d4efbeb11dbab97c38cf2b9c115b00a2ccc6752c3e2d552a3b"}
Sep 16 19:17:16.596272 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:16.596217    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv" podStartSLOduration=1.59620448 podStartE2EDuration="1.59620448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:16.59539499 +0000 UTC m=+2274.954912984" watchObservedRunningTime="2026-09-16 19:17:16.59620448 +0000 UTC m=+2274.955722475"
Sep 16 19:17:16.716982 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:16.716953    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:25.717095 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:25.717059    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:25.729462 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:25.729439    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:17:26.637145 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:17:26.637112    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:18:05.765451 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:05.765415    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"]
Sep 16 19:18:20.784603 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:20.784496    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv" podUID="aa415c9d-ec09-493d-b9ce-6712f79911e3" containerName="main" containerID="cri-o://78e015a32b4de008da272dfcf641241c18a4a3d2bbdd888de91808cac7720a0c" gracePeriod=45
Sep 16 19:18:20.918925 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:20.918898    2565 generic.go:356] "Generic (PLEG): container finished" podID="aa415c9d-ec09-493d-b9ce-6712f79911e3" containerID="78e015a32b4de008da272dfcf641241c18a4a3d2bbdd888de91808cac7720a0c" exitCode=0
Sep 16 19:18:20.919040 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:20.918965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv" event={"ID":"aa415c9d-ec09-493d-b9ce-6712f79911e3","Type":"ContainerDied","Data":"78e015a32b4de008da272dfcf641241c18a4a3d2bbdd888de91808cac7720a0c"}
Sep 16 19:18:21.035812 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.035761    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:18:21.098870 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.098842    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-model-cache\") pod \"aa415c9d-ec09-493d-b9ce-6712f79911e3\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") "
Sep 16 19:18:21.099024 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.098882    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5nbl\" (UniqueName: \"kubernetes.io/projected/aa415c9d-ec09-493d-b9ce-6712f79911e3-kube-api-access-l5nbl\") pod \"aa415c9d-ec09-493d-b9ce-6712f79911e3\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") "
Sep 16 19:18:21.099024 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.098923    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-tmp-dir\") pod \"aa415c9d-ec09-493d-b9ce-6712f79911e3\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") "
Sep 16 19:18:21.099024 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.098985    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-dshm\") pod \"aa415c9d-ec09-493d-b9ce-6712f79911e3\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") "
Sep 16 19:18:21.099190 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.099044    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-home\") pod \"aa415c9d-ec09-493d-b9ce-6712f79911e3\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") "
Sep 16 19:18:21.099190 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.099094    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa415c9d-ec09-493d-b9ce-6712f79911e3-tls-certs\") pod \"aa415c9d-ec09-493d-b9ce-6712f79911e3\" (UID: \"aa415c9d-ec09-493d-b9ce-6712f79911e3\") "
Sep 16 19:18:21.099190 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.099135    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa415c9d-ec09-493d-b9ce-6712f79911e3" (UID: "aa415c9d-ec09-493d-b9ce-6712f79911e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:21.099340 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.099213    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa415c9d-ec09-493d-b9ce-6712f79911e3" (UID: "aa415c9d-ec09-493d-b9ce-6712f79911e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:21.099340 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.099323    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-home" (OuterVolumeSpecName: "home") pod "aa415c9d-ec09-493d-b9ce-6712f79911e3" (UID: "aa415c9d-ec09-493d-b9ce-6712f79911e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:21.099467 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.099400    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:18:21.099467 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.099412    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:18:21.099467 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.099422    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:18:21.101126 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.101107    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-dshm" (OuterVolumeSpecName: "dshm") pod "aa415c9d-ec09-493d-b9ce-6712f79911e3" (UID: "aa415c9d-ec09-493d-b9ce-6712f79911e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:21.101227 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.101153    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa415c9d-ec09-493d-b9ce-6712f79911e3-kube-api-access-l5nbl" (OuterVolumeSpecName: "kube-api-access-l5nbl") pod "aa415c9d-ec09-493d-b9ce-6712f79911e3" (UID: "aa415c9d-ec09-493d-b9ce-6712f79911e3"). InnerVolumeSpecName "kube-api-access-l5nbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:21.101227 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.101223    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa415c9d-ec09-493d-b9ce-6712f79911e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa415c9d-ec09-493d-b9ce-6712f79911e3" (UID: "aa415c9d-ec09-493d-b9ce-6712f79911e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:21.200778 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.200738    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa415c9d-ec09-493d-b9ce-6712f79911e3-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:18:21.200778 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.200778    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5nbl\" (UniqueName: \"kubernetes.io/projected/aa415c9d-ec09-493d-b9ce-6712f79911e3-kube-api-access-l5nbl\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:18:21.200917 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.200793    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa415c9d-ec09-493d-b9ce-6712f79911e3-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:18:21.926596 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.926560    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"
Sep 16 19:18:21.926596 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.926570    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv" event={"ID":"aa415c9d-ec09-493d-b9ce-6712f79911e3","Type":"ContainerDied","Data":"df5bb6a2561f80d4efbeb11dbab97c38cf2b9c115b00a2ccc6752c3e2d552a3b"}
Sep 16 19:18:21.927079 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.926633    2565 scope.go:117] "RemoveContainer" containerID="78e015a32b4de008da272dfcf641241c18a4a3d2bbdd888de91808cac7720a0c"
Sep 16 19:18:21.974669 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.974637    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"]
Sep 16 19:18:21.979626 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.979600    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-ckfpv"]
Sep 16 19:18:21.995997 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:21.995967    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"]
Sep 16 19:18:22.208201 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:22.208124    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa415c9d-ec09-493d-b9ce-6712f79911e3" path="/var/lib/kubelet/pods/aa415c9d-ec09-493d-b9ce-6712f79911e3/volumes"
Sep 16 19:18:37.015689 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.015627    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl" podUID="c848259e-7f7b-4b41-b43b-ad537df2a11a" containerName="main" containerID="cri-o://48dd7372689f414af7514f51c3ae00dc6bcf1450872484ab47c07ad0499f123d" gracePeriod=45
Sep 16 19:18:37.271160 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.271106    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:18:37.324887 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.324863    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c848259e-7f7b-4b41-b43b-ad537df2a11a-tls-certs\") pod \"c848259e-7f7b-4b41-b43b-ad537df2a11a\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") "
Sep 16 19:18:37.325042 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.324923    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-model-cache\") pod \"c848259e-7f7b-4b41-b43b-ad537df2a11a\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") "
Sep 16 19:18:37.325042 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.324942    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-dshm\") pod \"c848259e-7f7b-4b41-b43b-ad537df2a11a\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") "
Sep 16 19:18:37.325042 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.324967    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-home\") pod \"c848259e-7f7b-4b41-b43b-ad537df2a11a\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") "
Sep 16 19:18:37.325042 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.324999    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n5cs\" (UniqueName: \"kubernetes.io/projected/c848259e-7f7b-4b41-b43b-ad537df2a11a-kube-api-access-2n5cs\") pod \"c848259e-7f7b-4b41-b43b-ad537df2a11a\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") "
Sep 16 19:18:37.325042 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.325034    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-tmp-dir\") pod \"c848259e-7f7b-4b41-b43b-ad537df2a11a\" (UID: \"c848259e-7f7b-4b41-b43b-ad537df2a11a\") "
Sep 16 19:18:37.325363 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.325204    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-home" (OuterVolumeSpecName: "home") pod "c848259e-7f7b-4b41-b43b-ad537df2a11a" (UID: "c848259e-7f7b-4b41-b43b-ad537df2a11a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:37.325363 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.325220    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-model-cache" (OuterVolumeSpecName: "model-cache") pod "c848259e-7f7b-4b41-b43b-ad537df2a11a" (UID: "c848259e-7f7b-4b41-b43b-ad537df2a11a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:37.325477 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.325370    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c848259e-7f7b-4b41-b43b-ad537df2a11a" (UID: "c848259e-7f7b-4b41-b43b-ad537df2a11a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:37.325477 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.325424    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:18:37.325477 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.325442    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:18:37.326949 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.326925    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c848259e-7f7b-4b41-b43b-ad537df2a11a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c848259e-7f7b-4b41-b43b-ad537df2a11a" (UID: "c848259e-7f7b-4b41-b43b-ad537df2a11a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:37.327046 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.326999    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-dshm" (OuterVolumeSpecName: "dshm") pod "c848259e-7f7b-4b41-b43b-ad537df2a11a" (UID: "c848259e-7f7b-4b41-b43b-ad537df2a11a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:37.327209 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.327190    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c848259e-7f7b-4b41-b43b-ad537df2a11a-kube-api-access-2n5cs" (OuterVolumeSpecName: "kube-api-access-2n5cs") pod "c848259e-7f7b-4b41-b43b-ad537df2a11a" (UID: "c848259e-7f7b-4b41-b43b-ad537df2a11a"). InnerVolumeSpecName "kube-api-access-2n5cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:37.426730 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.426705    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:18:37.426730 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.426727    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c848259e-7f7b-4b41-b43b-ad537df2a11a-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:18:37.426855 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.426736    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c848259e-7f7b-4b41-b43b-ad537df2a11a-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:18:37.426855 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:37.426744    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n5cs\" (UniqueName: \"kubernetes.io/projected/c848259e-7f7b-4b41-b43b-ad537df2a11a-kube-api-access-2n5cs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:18:38.012467 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:38.012434    2565 generic.go:356] "Generic (PLEG): container finished" podID="c848259e-7f7b-4b41-b43b-ad537df2a11a" containerID="48dd7372689f414af7514f51c3ae00dc6bcf1450872484ab47c07ad0499f123d" exitCode=0
Sep 16 19:18:38.012639 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:38.012480    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"
Sep 16 19:18:38.012639 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:38.012520    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl" event={"ID":"c848259e-7f7b-4b41-b43b-ad537df2a11a","Type":"ContainerDied","Data":"48dd7372689f414af7514f51c3ae00dc6bcf1450872484ab47c07ad0499f123d"}
Sep 16 19:18:38.012639 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:38.012625    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl" event={"ID":"c848259e-7f7b-4b41-b43b-ad537df2a11a","Type":"ContainerDied","Data":"c858aa28b9ab9eff82e5415de750b48e44011f200e8bfa2c6cd6ec777042ba2b"}
Sep 16 19:18:38.012828 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:38.012651    2565 scope.go:117] "RemoveContainer" containerID="48dd7372689f414af7514f51c3ae00dc6bcf1450872484ab47c07ad0499f123d"
Sep 16 19:18:38.022073 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:38.021937    2565 scope.go:117] "RemoveContainer" containerID="48dd7372689f414af7514f51c3ae00dc6bcf1450872484ab47c07ad0499f123d"
Sep 16 19:18:38.022269 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:18:38.022200    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48dd7372689f414af7514f51c3ae00dc6bcf1450872484ab47c07ad0499f123d\": container with ID starting with 48dd7372689f414af7514f51c3ae00dc6bcf1450872484ab47c07ad0499f123d not found: ID does not exist" containerID="48dd7372689f414af7514f51c3ae00dc6bcf1450872484ab47c07ad0499f123d"
Sep 16 19:18:38.022269 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:38.022228    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48dd7372689f414af7514f51c3ae00dc6bcf1450872484ab47c07ad0499f123d"} err="failed to get container status \"48dd7372689f414af7514f51c3ae00dc6bcf1450872484ab47c07ad0499f123d\": rpc error: code = NotFound desc = could not find container \"48dd7372689f414af7514f51c3ae00dc6bcf1450872484ab47c07ad0499f123d\": container with ID starting with 48dd7372689f414af7514f51c3ae00dc6bcf1450872484ab47c07ad0499f123d not found: ID does not exist"
Sep 16 19:18:38.047237 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:38.047215    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"]
Sep 16 19:18:38.063845 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:38.063815    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-qhdvl"]
Sep 16 19:18:38.206911 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:18:38.206889    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c848259e-7f7b-4b41-b43b-ad537df2a11a" path="/var/lib/kubelet/pods/c848259e-7f7b-4b41-b43b-ad537df2a11a/volumes"
Sep 16 19:21:31.617782 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.617747    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"]
Sep 16 19:21:31.618210 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.618123    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa415c9d-ec09-493d-b9ce-6712f79911e3" containerName="main"
Sep 16 19:21:31.618210 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.618136    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa415c9d-ec09-493d-b9ce-6712f79911e3" containerName="main"
Sep 16 19:21:31.618210 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.618154    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c848259e-7f7b-4b41-b43b-ad537df2a11a" containerName="main"
Sep 16 19:21:31.618210 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.618160    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c848259e-7f7b-4b41-b43b-ad537df2a11a" containerName="main"
Sep 16 19:21:31.618377 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.618222    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa415c9d-ec09-493d-b9ce-6712f79911e3" containerName="main"
Sep 16 19:21:31.618377 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.618231    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c848259e-7f7b-4b41-b43b-ad537df2a11a" containerName="main"
Sep 16 19:21:31.621252 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.621234    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.628653 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.628630    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 16 19:21:31.630671 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.630653    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-pxsd4\""
Sep 16 19:21:31.656966 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.656944    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 16 19:21:31.657081 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.657000    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:21:31.660616 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.660594    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"]
Sep 16 19:21:31.708557 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.708529    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-envoy\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.708682 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.708563    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-data\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.708682 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.708585    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.708682 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.708646    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7k5p\" (UniqueName: \"kubernetes.io/projected/c2ced697-708d-4147-a17a-f7b3fbd1dd79-kube-api-access-k7k5p\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.708828 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.708700    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-workload-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.708828 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.708716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-token\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.708828 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.708746    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-credential-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.708828 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.708815    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.708968 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.708856    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-workload-certs\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.809877 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.809813    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7k5p\" (UniqueName: \"kubernetes.io/projected/c2ced697-708d-4147-a17a-f7b3fbd1dd79-kube-api-access-k7k5p\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.809877 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.809849    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-workload-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.809877 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.809867    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-token\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.810119 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.809984    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-credential-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.810119 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.810026    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.810119 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.810062    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-workload-certs\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.810119 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.810087    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-envoy\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.810364 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.810120    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-data\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.810364 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.810154    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.810364 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.810249    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-workload-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.810499 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.810370    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-credential-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.810536 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.810521    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-data\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.810571 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.810552    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-workload-certs\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.810856 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.810838    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.812435 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.812417    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-envoy\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.812584 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.812567    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.835813 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.835793    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7k5p\" (UniqueName: \"kubernetes.io/projected/c2ced697-708d-4147-a17a-f7b3fbd1dd79-kube-api-access-k7k5p\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.836026 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.836009    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-token\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-w495c\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:31.931112 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:31.931091    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:32.251994 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:32.251964    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"]
Sep 16 19:21:32.253618 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:21:32.253590    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ced697_708d_4147_a17a_f7b3fbd1dd79.slice/crio-a79921fada8c370095fe030dae160c99472c5d18606b717c4667b0dfd4c9fba0 WatchSource:0}: Error finding container a79921fada8c370095fe030dae160c99472c5d18606b717c4667b0dfd4c9fba0: Status 404 returned error can't find the container with id a79921fada8c370095fe030dae160c99472c5d18606b717c4667b0dfd4c9fba0
Sep 16 19:21:32.255539 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:32.255521    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:21:32.941246 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:32.941211    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c" event={"ID":"c2ced697-708d-4147-a17a-f7b3fbd1dd79","Type":"ContainerStarted","Data":"a79921fada8c370095fe030dae160c99472c5d18606b717c4667b0dfd4c9fba0"}
Sep 16 19:21:35.522274 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:35.522230    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:21:35.522564 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:35.522320    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:21:35.522564 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:35.522366    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:21:35.962294 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:35.962205    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c" event={"ID":"c2ced697-708d-4147-a17a-f7b3fbd1dd79","Type":"ContainerStarted","Data":"f17ec7baf1be269d5ebe607f1cad51f806aa8a227d481586abbe35782f392541"}
Sep 16 19:21:36.017638 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:36.017572    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c" podStartSLOduration=1.7512191609999999 podStartE2EDuration="5.017556783s" totalImagesPullingTime="3.266337622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:36.017089277 +0000 UTC m=+2534.376607276" watchObservedRunningTime="2026-09-16 19:21:36.017556783 +0000 UTC m=+2534.377074774"
Sep 16 19:21:36.931713 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:36.931673    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:36.936516 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:36.936489    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:36.967287 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:36.967262    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:36.968040 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:36.968016    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:21:49.903196 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:49.903165    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"]
Sep 16 19:21:49.906925 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:49.906905    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:49.915775 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:49.915752    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 19:21:49.915871 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:49.915752    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 16 19:21:49.915871 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:49.915850    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 19:21:49.915959 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:49.915827    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-vhfcs\""
Sep 16 19:21:49.923685 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:49.923659    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"]
Sep 16 19:21:50.053483 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.053451    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:50.053642 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.053492    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:50.053642 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.053622    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:50.053725 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.053652    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:50.053725 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.053668    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsmhb\" (UniqueName: \"kubernetes.io/projected/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-kube-api-access-rsmhb\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:50.053725 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.053714    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:50.154379 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.154290    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:50.154379 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.154323    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:50.154379 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.154355    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsmhb\" (UniqueName: \"kubernetes.io/projected/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-kube-api-access-rsmhb\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:50.154644 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.154394    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:50.154644 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.154434    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:50.154644 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.154466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:50.154801 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.154771    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:50.154865 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.154838    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:50.154919 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.154835    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:50.156663 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.156643    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:50.156855 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.156838    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:50.176202 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.176180    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsmhb\" (UniqueName: \"kubernetes.io/projected/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-kube-api-access-rsmhb\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:50.217688 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.217658    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:21:50.498998 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:50.498964    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"]
Sep 16 19:21:50.500545 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:21:50.500519    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5a8b3e_4f25_4922_bf7c_4f3c736ff2aa.slice/crio-e8da9503476032aa4c8efe65560f6963d5cd06827547726b5294846d2b6db00a WatchSource:0}: Error finding container e8da9503476032aa4c8efe65560f6963d5cd06827547726b5294846d2b6db00a: Status 404 returned error can't find the container with id e8da9503476032aa4c8efe65560f6963d5cd06827547726b5294846d2b6db00a
Sep 16 19:21:51.044892 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:51.044859    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t" event={"ID":"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa","Type":"ContainerStarted","Data":"2566c632fe5f2455434306a7fa04c48cb128019abe2a7f711f3c9cc7dc4fb539"}
Sep 16 19:21:51.045317 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:51.044919    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t" event={"ID":"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa","Type":"ContainerStarted","Data":"e8da9503476032aa4c8efe65560f6963d5cd06827547726b5294846d2b6db00a"}
Sep 16 19:21:51.218535 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:21:51.218498    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:22:00.218134 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:00.218108    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:22:00.229998 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:00.229977    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:22:00.253914 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:00.253868    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t" podStartSLOduration=11.253855657999999 podStartE2EDuration="11.253855658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:51.074468181 +0000 UTC m=+2549.433986175" watchObservedRunningTime="2026-09-16 19:22:00.253855658 +0000 UTC m=+2558.613373652"
Sep 16 19:22:01.112148 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:01.112116    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:22:16.185842 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:16.185804    2565 generic.go:356] "Generic (PLEG): container finished" podID="c2ced697-708d-4147-a17a-f7b3fbd1dd79" containerID="f17ec7baf1be269d5ebe607f1cad51f806aa8a227d481586abbe35782f392541" exitCode=0
Sep 16 19:22:16.186248 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:16.185884    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c" event={"ID":"c2ced697-708d-4147-a17a-f7b3fbd1dd79","Type":"ContainerDied","Data":"f17ec7baf1be269d5ebe607f1cad51f806aa8a227d481586abbe35782f392541"}
Sep 16 19:22:16.186381 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:16.186366    2565 scope.go:117] "RemoveContainer" containerID="f17ec7baf1be269d5ebe607f1cad51f806aa8a227d481586abbe35782f392541"
Sep 16 19:22:16.187435 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:16.187407    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:22:16.187515 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:16.187483    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:22:16.187557 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:16.187540    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:22:17.201302 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:17.200202    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c" event={"ID":"c2ced697-708d-4147-a17a-f7b3fbd1dd79","Type":"ContainerStarted","Data":"40d07442bed3e6b8b734d96b461a46a8f0e8d8ce855422ae453b08c33bccd8a2"}
Sep 16 19:22:17.931213 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:17.931179    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:22:17.932542 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:17.932512    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c" podUID="c2ced697-708d-4147-a17a-f7b3fbd1dd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 19:22:18.931630 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:18.931583    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c" podUID="c2ced697-708d-4147-a17a-f7b3fbd1dd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 19:22:19.931617 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:19.931577    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c" podUID="c2ced697-708d-4147-a17a-f7b3fbd1dd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 19:22:20.932451 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:20.932409    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c" podUID="c2ced697-708d-4147-a17a-f7b3fbd1dd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 19:22:21.931445 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:21.931401    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c" podUID="c2ced697-708d-4147-a17a-f7b3fbd1dd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 19:22:21.967656 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:21.967633    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:22:22.935992 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:22.935942    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:22:22.936957 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:22.936939    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:22:28.387456 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.387418    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"]
Sep 16 19:22:28.398669 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:22:28.398633    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:22:28.398669 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:22:28.398669 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 19:22:28.398669 ip-10-0-130-204 kubenswrapper[2565]:  > containerID="2566c632fe5f2455434306a7fa04c48cb128019abe2a7f711f3c9cc7dc4fb539" cmd=["/bin/sleep","15"]
Sep 16 19:22:28.398977 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.398689    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t" message="" err=<
Sep 16 19:22:28.398977 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:22:28.398977 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 19:22:28.398977 ip-10-0-130-204 kubenswrapper[2565]:  >
Sep 16 19:22:28.398977 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:22:28.398701    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:22:28.398977 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:22:28.398977 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 19:22:28.398977 ip-10-0-130-204 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t" podUID="ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa" containerName="main" containerID="cri-o://2566c632fe5f2455434306a7fa04c48cb128019abe2a7f711f3c9cc7dc4fb539"
Sep 16 19:22:28.398977 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.398743    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t" podUID="ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa" containerName="main" containerID="cri-o://2566c632fe5f2455434306a7fa04c48cb128019abe2a7f711f3c9cc7dc4fb539" gracePeriod=60
Sep 16 19:22:28.754403 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.754378    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:22:28.783940 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.783912    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsmhb\" (UniqueName: \"kubernetes.io/projected/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-kube-api-access-rsmhb\") pod \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") "
Sep 16 19:22:28.784065 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.783969    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-dshm\") pod \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") "
Sep 16 19:22:28.784065 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.783993    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-model-cache\") pod \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") "
Sep 16 19:22:28.784748 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.784401    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-home\") pod \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") "
Sep 16 19:22:28.784748 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.784413    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa" (UID: "ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:28.784748 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.784479    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-tmp-dir\") pod \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") "
Sep 16 19:22:28.784748 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.784570    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-tls-certs\") pod \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\" (UID: \"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa\") "
Sep 16 19:22:28.784748 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.784694    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-home" (OuterVolumeSpecName: "home") pod "ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa" (UID: "ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:28.784748 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.784712    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa" (UID: "ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:28.785117 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.784927    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:22:28.785117 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.784945    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:22:28.785117 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.784957    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:22:28.787374 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.787314    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-kube-api-access-rsmhb" (OuterVolumeSpecName: "kube-api-access-rsmhb") pod "ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa" (UID: "ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa"). InnerVolumeSpecName "kube-api-access-rsmhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:28.787502 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.787482    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa" (UID: "ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:28.787808 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.787787    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-dshm" (OuterVolumeSpecName: "dshm") pod "ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa" (UID: "ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:28.886634 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.886571    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:22:28.886634 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.886633    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsmhb\" (UniqueName: \"kubernetes.io/projected/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-kube-api-access-rsmhb\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:22:28.886906 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:28.886656    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:22:29.275203 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:29.275170    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa" containerID="2566c632fe5f2455434306a7fa04c48cb128019abe2a7f711f3c9cc7dc4fb539" exitCode=0
Sep 16 19:22:29.275401 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:29.275211    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"
Sep 16 19:22:29.275401 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:29.275290    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t" event={"ID":"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa","Type":"ContainerDied","Data":"2566c632fe5f2455434306a7fa04c48cb128019abe2a7f711f3c9cc7dc4fb539"}
Sep 16 19:22:29.275401 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:29.275339    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t" event={"ID":"ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa","Type":"ContainerDied","Data":"e8da9503476032aa4c8efe65560f6963d5cd06827547726b5294846d2b6db00a"}
Sep 16 19:22:29.275401 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:29.275390    2565 scope.go:117] "RemoveContainer" containerID="2566c632fe5f2455434306a7fa04c48cb128019abe2a7f711f3c9cc7dc4fb539"
Sep 16 19:22:29.286366 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:29.286327    2565 scope.go:117] "RemoveContainer" containerID="2566c632fe5f2455434306a7fa04c48cb128019abe2a7f711f3c9cc7dc4fb539"
Sep 16 19:22:29.286742 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:22:29.286713    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2566c632fe5f2455434306a7fa04c48cb128019abe2a7f711f3c9cc7dc4fb539\": container with ID starting with 2566c632fe5f2455434306a7fa04c48cb128019abe2a7f711f3c9cc7dc4fb539 not found: ID does not exist" containerID="2566c632fe5f2455434306a7fa04c48cb128019abe2a7f711f3c9cc7dc4fb539"
Sep 16 19:22:29.286853 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:29.286754    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2566c632fe5f2455434306a7fa04c48cb128019abe2a7f711f3c9cc7dc4fb539"} err="failed to get container status \"2566c632fe5f2455434306a7fa04c48cb128019abe2a7f711f3c9cc7dc4fb539\": rpc error: code = NotFound desc = could not find container \"2566c632fe5f2455434306a7fa04c48cb128019abe2a7f711f3c9cc7dc4fb539\": container with ID starting with 2566c632fe5f2455434306a7fa04c48cb128019abe2a7f711f3c9cc7dc4fb539 not found: ID does not exist"
Sep 16 19:22:29.300792 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:29.300765    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"]
Sep 16 19:22:29.306379 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:29.306357    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995z742t"]
Sep 16 19:22:30.207412 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:30.207377    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa" path="/var/lib/kubelet/pods/ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa/volumes"
Sep 16 19:22:38.044568 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:38.044530    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"]
Sep 16 19:22:38.045013 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:38.044741    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c" podUID="c2ced697-708d-4147-a17a-f7b3fbd1dd79" containerName="istio-proxy" containerID="cri-o://40d07442bed3e6b8b734d96b461a46a8f0e8d8ce855422ae453b08c33bccd8a2" gracePeriod=30
Sep 16 19:22:43.299522 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.299486    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:22:43.350630 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.350545    2565 generic.go:356] "Generic (PLEG): container finished" podID="c2ced697-708d-4147-a17a-f7b3fbd1dd79" containerID="40d07442bed3e6b8b734d96b461a46a8f0e8d8ce855422ae453b08c33bccd8a2" exitCode=0
Sep 16 19:22:43.350630 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.350591    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"
Sep 16 19:22:43.350799 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.350624    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c" event={"ID":"c2ced697-708d-4147-a17a-f7b3fbd1dd79","Type":"ContainerDied","Data":"40d07442bed3e6b8b734d96b461a46a8f0e8d8ce855422ae453b08c33bccd8a2"}
Sep 16 19:22:43.350799 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.350678    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c" event={"ID":"c2ced697-708d-4147-a17a-f7b3fbd1dd79","Type":"ContainerDied","Data":"a79921fada8c370095fe030dae160c99472c5d18606b717c4667b0dfd4c9fba0"}
Sep 16 19:22:43.350799 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.350702    2565 scope.go:117] "RemoveContainer" containerID="40d07442bed3e6b8b734d96b461a46a8f0e8d8ce855422ae453b08c33bccd8a2"
Sep 16 19:22:43.363255 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.363234    2565 scope.go:117] "RemoveContainer" containerID="f17ec7baf1be269d5ebe607f1cad51f806aa8a227d481586abbe35782f392541"
Sep 16 19:22:43.373037 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.373015    2565 scope.go:117] "RemoveContainer" containerID="40d07442bed3e6b8b734d96b461a46a8f0e8d8ce855422ae453b08c33bccd8a2"
Sep 16 19:22:43.373392 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:22:43.373367    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d07442bed3e6b8b734d96b461a46a8f0e8d8ce855422ae453b08c33bccd8a2\": container with ID starting with 40d07442bed3e6b8b734d96b461a46a8f0e8d8ce855422ae453b08c33bccd8a2 not found: ID does not exist" containerID="40d07442bed3e6b8b734d96b461a46a8f0e8d8ce855422ae453b08c33bccd8a2"
Sep 16 19:22:43.373503 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.373404    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d07442bed3e6b8b734d96b461a46a8f0e8d8ce855422ae453b08c33bccd8a2"} err="failed to get container status \"40d07442bed3e6b8b734d96b461a46a8f0e8d8ce855422ae453b08c33bccd8a2\": rpc error: code = NotFound desc = could not find container \"40d07442bed3e6b8b734d96b461a46a8f0e8d8ce855422ae453b08c33bccd8a2\": container with ID starting with 40d07442bed3e6b8b734d96b461a46a8f0e8d8ce855422ae453b08c33bccd8a2 not found: ID does not exist"
Sep 16 19:22:43.373503 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.373427    2565 scope.go:117] "RemoveContainer" containerID="f17ec7baf1be269d5ebe607f1cad51f806aa8a227d481586abbe35782f392541"
Sep 16 19:22:43.373776 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:22:43.373756    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f17ec7baf1be269d5ebe607f1cad51f806aa8a227d481586abbe35782f392541\": container with ID starting with f17ec7baf1be269d5ebe607f1cad51f806aa8a227d481586abbe35782f392541 not found: ID does not exist" containerID="f17ec7baf1be269d5ebe607f1cad51f806aa8a227d481586abbe35782f392541"
Sep 16 19:22:43.373843 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.373786    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17ec7baf1be269d5ebe607f1cad51f806aa8a227d481586abbe35782f392541"} err="failed to get container status \"f17ec7baf1be269d5ebe607f1cad51f806aa8a227d481586abbe35782f392541\": rpc error: code = NotFound desc = could not find container \"f17ec7baf1be269d5ebe607f1cad51f806aa8a227d481586abbe35782f392541\": container with ID starting with f17ec7baf1be269d5ebe607f1cad51f806aa8a227d481586abbe35782f392541 not found: ID does not exist"
Sep 16 19:22:43.398121 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.398094    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istiod-ca-cert\") pod \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") "
Sep 16 19:22:43.398227 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.398144    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-credential-socket\") pod \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") "
Sep 16 19:22:43.398227 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.398180    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-podinfo\") pod \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") "
Sep 16 19:22:43.398227 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.398218    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7k5p\" (UniqueName: \"kubernetes.io/projected/c2ced697-708d-4147-a17a-f7b3fbd1dd79-kube-api-access-k7k5p\") pod \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") "
Sep 16 19:22:43.398430 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.398265    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-data\") pod \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") "
Sep 16 19:22:43.398489 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.398411    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-workload-socket\") pod \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") "
Sep 16 19:22:43.398547 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.398499    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c2ced697-708d-4147-a17a-f7b3fbd1dd79" (UID: "c2ced697-708d-4147-a17a-f7b3fbd1dd79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:43.398547 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.398505    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-workload-certs\") pod \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") "
Sep 16 19:22:43.398654 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.398585    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-token\") pod \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") "
Sep 16 19:22:43.398654 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.398644    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-envoy\") pod \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\" (UID: \"c2ced697-708d-4147-a17a-f7b3fbd1dd79\") "
Sep 16 19:22:43.398752 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.398676    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c2ced697-708d-4147-a17a-f7b3fbd1dd79" (UID: "c2ced697-708d-4147-a17a-f7b3fbd1dd79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:43.398752 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.398700    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c2ced697-708d-4147-a17a-f7b3fbd1dd79" (UID: "c2ced697-708d-4147-a17a-f7b3fbd1dd79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:43.398903 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.398881    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-data" (OuterVolumeSpecName: "istio-data") pod "c2ced697-708d-4147-a17a-f7b3fbd1dd79" (UID: "c2ced697-708d-4147-a17a-f7b3fbd1dd79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:43.398959 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.398941    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-workload-socket\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:22:43.399010 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.398962    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-workload-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:22:43.399010 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.398977    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-credential-socket\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:22:43.399185 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.399161    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c2ced697-708d-4147-a17a-f7b3fbd1dd79" (UID: "c2ced697-708d-4147-a17a-f7b3fbd1dd79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:43.400937 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.400891    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ced697-708d-4147-a17a-f7b3fbd1dd79-kube-api-access-k7k5p" (OuterVolumeSpecName: "kube-api-access-k7k5p") pod "c2ced697-708d-4147-a17a-f7b3fbd1dd79" (UID: "c2ced697-708d-4147-a17a-f7b3fbd1dd79"). InnerVolumeSpecName "kube-api-access-k7k5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:43.401031 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.401014    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c2ced697-708d-4147-a17a-f7b3fbd1dd79" (UID: "c2ced697-708d-4147-a17a-f7b3fbd1dd79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:22:43.401031 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.401020    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-token" (OuterVolumeSpecName: "istio-token") pod "c2ced697-708d-4147-a17a-f7b3fbd1dd79" (UID: "c2ced697-708d-4147-a17a-f7b3fbd1dd79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:43.401387 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.401362    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c2ced697-708d-4147-a17a-f7b3fbd1dd79" (UID: "c2ced697-708d-4147-a17a-f7b3fbd1dd79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:43.499688 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.499660    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-token\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:22:43.499688 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.499685    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-envoy\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:22:43.499837 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.499694    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istiod-ca-cert\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:22:43.499837 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.499702    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-podinfo\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:22:43.499837 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.499711    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7k5p\" (UniqueName: \"kubernetes.io/projected/c2ced697-708d-4147-a17a-f7b3fbd1dd79-kube-api-access-k7k5p\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:22:43.499837 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.499720    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ced697-708d-4147-a17a-f7b3fbd1dd79-istio-data\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:22:43.675213 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.675184    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"]
Sep 16 19:22:43.677600 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:43.677574    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-w495c"]
Sep 16 19:22:44.207738 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:22:44.207668    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2ced697-708d-4147-a17a-f7b3fbd1dd79" path="/var/lib/kubelet/pods/c2ced697-708d-4147-a17a-f7b3fbd1dd79/volumes"
Sep 16 19:25:00.871700 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:00.871662    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"]
Sep 16 19:25:00.872114 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:00.872035    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ced697-708d-4147-a17a-f7b3fbd1dd79" containerName="istio-proxy"
Sep 16 19:25:00.872114 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:00.872047    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ced697-708d-4147-a17a-f7b3fbd1dd79" containerName="istio-proxy"
Sep 16 19:25:00.872114 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:00.872062    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa" containerName="main"
Sep 16 19:25:00.872114 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:00.872068    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa" containerName="main"
Sep 16 19:25:00.872254 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:00.872138    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2ced697-708d-4147-a17a-f7b3fbd1dd79" containerName="istio-proxy"
Sep 16 19:25:00.872254 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:00.872146    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2ced697-708d-4147-a17a-f7b3fbd1dd79" containerName="istio-proxy"
Sep 16 19:25:00.872254 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:00.872154    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad5a8b3e-4f25-4922-bf7c-4f3c736ff2aa" containerName="main"
Sep 16 19:25:00.875153 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:00.875136    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:00.878095 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:00.878062    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 19:25:00.878095 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:00.878067    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-jmlm9\""
Sep 16 19:25:00.879083 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:00.879060    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 19:25:00.879243 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:00.879107    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 16 19:25:00.897029 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:00.896999    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"]
Sep 16 19:25:01.018410 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.018374    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:01.018410 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.018410    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:01.018658 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.018434    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc314259-10cc-47a2-905c-f3096534e637-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:01.018658 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.018464    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r96h9\" (UniqueName: \"kubernetes.io/projected/fc314259-10cc-47a2-905c-f3096534e637-kube-api-access-r96h9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:01.018658 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.018523    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:01.018658 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.018557    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:01.119662 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.119635    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:01.119816 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.119665    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:01.119816 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.119682    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc314259-10cc-47a2-905c-f3096534e637-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:01.119816 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.119703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r96h9\" (UniqueName: \"kubernetes.io/projected/fc314259-10cc-47a2-905c-f3096534e637-kube-api-access-r96h9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:01.119816 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.119724    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:01.119816 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.119740    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:01.120139 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.120113    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:01.120204 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.120132    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:01.120204 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.120169    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:01.122019 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.121963    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:01.122284 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.122266    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc314259-10cc-47a2-905c-f3096534e637-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:01.129265 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.129245    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r96h9\" (UniqueName: \"kubernetes.io/projected/fc314259-10cc-47a2-905c-f3096534e637-kube-api-access-r96h9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:01.185932 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.185906    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:01.467306 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:01.467281    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"]
Sep 16 19:25:01.470567 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:25:01.470540    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc314259_10cc_47a2_905c_f3096534e637.slice/crio-3e855e235326fe2f1021ee0e124d694749bb4bb16474f8a6ae2dd31b1fb198c4 WatchSource:0}: Error finding container 3e855e235326fe2f1021ee0e124d694749bb4bb16474f8a6ae2dd31b1fb198c4: Status 404 returned error can't find the container with id 3e855e235326fe2f1021ee0e124d694749bb4bb16474f8a6ae2dd31b1fb198c4
Sep 16 19:25:02.092760 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:02.092717    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt" event={"ID":"fc314259-10cc-47a2-905c-f3096534e637","Type":"ContainerStarted","Data":"6aa9955bead87a6309a9040c457667a30dab5f335e444c66020f7b56f3da7a57"}
Sep 16 19:25:02.093142 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:02.092783    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt" event={"ID":"fc314259-10cc-47a2-905c-f3096534e637","Type":"ContainerStarted","Data":"3e855e235326fe2f1021ee0e124d694749bb4bb16474f8a6ae2dd31b1fb198c4"}
Sep 16 19:25:02.115214 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:02.115162    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt" podStartSLOduration=2.115149418 podStartE2EDuration="2.115149418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:02.111963926 +0000 UTC m=+2740.471481920" watchObservedRunningTime="2026-09-16 19:25:02.115149418 +0000 UTC m=+2740.474667412"
Sep 16 19:25:02.186767 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:02.186731    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:11.186110 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:11.186076    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:11.198159 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:11.198133    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:12.156225 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:12.156192    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:34.436251 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.436209    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"]
Sep 16 19:25:34.445664 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:25:34.445628    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:25:34.445664 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:25:34.445664 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 19:25:34.445664 ip-10-0-130-204 kubenswrapper[2565]:  > containerID="6aa9955bead87a6309a9040c457667a30dab5f335e444c66020f7b56f3da7a57" cmd=["/bin/sleep","15"]
Sep 16 19:25:34.445951 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.445686    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt" message="" err=<
Sep 16 19:25:34.445951 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:25:34.445951 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 19:25:34.445951 ip-10-0-130-204 kubenswrapper[2565]:  >
Sep 16 19:25:34.445951 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:25:34.445703    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:25:34.445951 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:25:34.445951 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 19:25:34.445951 ip-10-0-130-204 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt" podUID="fc314259-10cc-47a2-905c-f3096534e637" containerName="main" containerID="cri-o://6aa9955bead87a6309a9040c457667a30dab5f335e444c66020f7b56f3da7a57"
Sep 16 19:25:34.445951 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.445757    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt" podUID="fc314259-10cc-47a2-905c-f3096534e637" containerName="main" containerID="cri-o://6aa9955bead87a6309a9040c457667a30dab5f335e444c66020f7b56f3da7a57" gracePeriod=60
Sep 16 19:25:34.793324 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.793298    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:34.884987 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.884952    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-tmp-dir\") pod \"fc314259-10cc-47a2-905c-f3096534e637\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") "
Sep 16 19:25:34.885147 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.884992    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-home\") pod \"fc314259-10cc-47a2-905c-f3096534e637\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") "
Sep 16 19:25:34.885147 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.885036    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-dshm\") pod \"fc314259-10cc-47a2-905c-f3096534e637\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") "
Sep 16 19:25:34.885147 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.885105    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc314259-10cc-47a2-905c-f3096534e637-tls-certs\") pod \"fc314259-10cc-47a2-905c-f3096534e637\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") "
Sep 16 19:25:34.885147 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.885141    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-model-cache\") pod \"fc314259-10cc-47a2-905c-f3096534e637\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") "
Sep 16 19:25:34.885388 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.885173    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r96h9\" (UniqueName: \"kubernetes.io/projected/fc314259-10cc-47a2-905c-f3096534e637-kube-api-access-r96h9\") pod \"fc314259-10cc-47a2-905c-f3096534e637\" (UID: \"fc314259-10cc-47a2-905c-f3096534e637\") "
Sep 16 19:25:34.885388 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.885204    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc314259-10cc-47a2-905c-f3096534e637" (UID: "fc314259-10cc-47a2-905c-f3096534e637"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:34.885388 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.885322    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-home" (OuterVolumeSpecName: "home") pod "fc314259-10cc-47a2-905c-f3096534e637" (UID: "fc314259-10cc-47a2-905c-f3096534e637"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:34.885556 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.885459    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc314259-10cc-47a2-905c-f3096534e637" (UID: "fc314259-10cc-47a2-905c-f3096534e637"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:34.885700 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.885675    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:25:34.885821 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.885704    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:25:34.885821 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.885717    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:25:34.887660 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.887632    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-dshm" (OuterVolumeSpecName: "dshm") pod "fc314259-10cc-47a2-905c-f3096534e637" (UID: "fc314259-10cc-47a2-905c-f3096534e637"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:34.887767 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.887654    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc314259-10cc-47a2-905c-f3096534e637-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc314259-10cc-47a2-905c-f3096534e637" (UID: "fc314259-10cc-47a2-905c-f3096534e637"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:34.887767 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.887653    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc314259-10cc-47a2-905c-f3096534e637-kube-api-access-r96h9" (OuterVolumeSpecName: "kube-api-access-r96h9") pod "fc314259-10cc-47a2-905c-f3096534e637" (UID: "fc314259-10cc-47a2-905c-f3096534e637"). InnerVolumeSpecName "kube-api-access-r96h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:34.986908 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.986877    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r96h9\" (UniqueName: \"kubernetes.io/projected/fc314259-10cc-47a2-905c-f3096534e637-kube-api-access-r96h9\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:25:34.987026 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.986909    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc314259-10cc-47a2-905c-f3096534e637-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:25:34.987026 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:34.986923    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc314259-10cc-47a2-905c-f3096534e637-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:25:35.268166 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:35.268075    2565 generic.go:356] "Generic (PLEG): container finished" podID="fc314259-10cc-47a2-905c-f3096534e637" containerID="6aa9955bead87a6309a9040c457667a30dab5f335e444c66020f7b56f3da7a57" exitCode=0
Sep 16 19:25:35.268166 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:35.268122    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"
Sep 16 19:25:35.268389 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:35.268164    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt" event={"ID":"fc314259-10cc-47a2-905c-f3096534e637","Type":"ContainerDied","Data":"6aa9955bead87a6309a9040c457667a30dab5f335e444c66020f7b56f3da7a57"}
Sep 16 19:25:35.268389 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:35.268223    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt" event={"ID":"fc314259-10cc-47a2-905c-f3096534e637","Type":"ContainerDied","Data":"3e855e235326fe2f1021ee0e124d694749bb4bb16474f8a6ae2dd31b1fb198c4"}
Sep 16 19:25:35.268389 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:35.268252    2565 scope.go:117] "RemoveContainer" containerID="6aa9955bead87a6309a9040c457667a30dab5f335e444c66020f7b56f3da7a57"
Sep 16 19:25:35.280175 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:35.280129    2565 scope.go:117] "RemoveContainer" containerID="6aa9955bead87a6309a9040c457667a30dab5f335e444c66020f7b56f3da7a57"
Sep 16 19:25:35.280617 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:25:35.280594    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa9955bead87a6309a9040c457667a30dab5f335e444c66020f7b56f3da7a57\": container with ID starting with 6aa9955bead87a6309a9040c457667a30dab5f335e444c66020f7b56f3da7a57 not found: ID does not exist" containerID="6aa9955bead87a6309a9040c457667a30dab5f335e444c66020f7b56f3da7a57"
Sep 16 19:25:35.280697 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:35.280625    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa9955bead87a6309a9040c457667a30dab5f335e444c66020f7b56f3da7a57"} err="failed to get container status \"6aa9955bead87a6309a9040c457667a30dab5f335e444c66020f7b56f3da7a57\": rpc error: code = NotFound desc = could not find container \"6aa9955bead87a6309a9040c457667a30dab5f335e444c66020f7b56f3da7a57\": container with ID starting with 6aa9955bead87a6309a9040c457667a30dab5f335e444c66020f7b56f3da7a57 not found: ID does not exist"
Sep 16 19:25:35.293876 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:35.293853    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"]
Sep 16 19:25:35.300538 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:35.300518    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-6zwzt"]
Sep 16 19:25:36.207124 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:36.207094    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc314259-10cc-47a2-905c-f3096534e637" path="/var/lib/kubelet/pods/fc314259-10cc-47a2-905c-f3096534e637/volumes"
Sep 16 19:25:40.651423 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.651391    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"]
Sep 16 19:25:40.651796 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.651779    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ced697-708d-4147-a17a-f7b3fbd1dd79" containerName="istio-proxy"
Sep 16 19:25:40.651796 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.651794    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ced697-708d-4147-a17a-f7b3fbd1dd79" containerName="istio-proxy"
Sep 16 19:25:40.651885 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.651815    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc314259-10cc-47a2-905c-f3096534e637" containerName="main"
Sep 16 19:25:40.651885 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.651821    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc314259-10cc-47a2-905c-f3096534e637" containerName="main"
Sep 16 19:25:40.651950 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.651892    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc314259-10cc-47a2-905c-f3096534e637" containerName="main"
Sep 16 19:25:40.656767 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.656735    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.659473 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.659446    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 16 19:25:40.659577 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.659553    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-jv2dq\""
Sep 16 19:25:40.659890 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.659866    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 16 19:25:40.659996 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.659928    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 16 19:25:40.668776 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.668744    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"]
Sep 16 19:25:40.733825 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.733795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9r9q\" (UniqueName: \"kubernetes.io/projected/8587aaaa-6e76-4930-afdc-8c28861106bb-kube-api-access-d9r9q\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.733953 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.733830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.733953 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.733872    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.733953 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.733912    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.734088 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.733963    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.734088 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.733981    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8587aaaa-6e76-4930-afdc-8c28861106bb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.734088 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.734007    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.835309 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.835282    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.835465 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.835331    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.835465 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.835367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.835576 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.835479    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.835576 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.835510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8587aaaa-6e76-4930-afdc-8c28861106bb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.835576 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.835533    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.835735 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.835590    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9r9q\" (UniqueName: \"kubernetes.io/projected/8587aaaa-6e76-4930-afdc-8c28861106bb-kube-api-access-d9r9q\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.835735 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.835705    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.835842 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.835816    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.835909 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.835888    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.835986 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.835969    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.837589 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.837569    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.837876 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.837858    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8587aaaa-6e76-4930-afdc-8c28861106bb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.849651 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.849628    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9r9q\" (UniqueName: \"kubernetes.io/projected/8587aaaa-6e76-4930-afdc-8c28861106bb-kube-api-access-d9r9q\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:40.968505 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:40.968428    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:25:41.250114 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:41.250088    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"]
Sep 16 19:25:41.301735 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:41.301707    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" event={"ID":"8587aaaa-6e76-4930-afdc-8c28861106bb","Type":"ContainerStarted","Data":"05aca42b7443c86af1ce16172d198ebb542fbd9bfd3f28a6cb99f436ab7fd0ea"}
Sep 16 19:25:48.345980 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:48.345941    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" event={"ID":"8587aaaa-6e76-4930-afdc-8c28861106bb","Type":"ContainerStarted","Data":"7f20aed0a13664a54af9bc5a361992c07b4bf4624f39720db04f5aa7de70cfa5"}
Sep 16 19:25:49.352637 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:49.352603    2565 generic.go:356] "Generic (PLEG): container finished" podID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerID="7f20aed0a13664a54af9bc5a361992c07b4bf4624f39720db04f5aa7de70cfa5" exitCode=0
Sep 16 19:25:49.352995 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:25:49.352662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" event={"ID":"8587aaaa-6e76-4930-afdc-8c28861106bb","Type":"ContainerDied","Data":"7f20aed0a13664a54af9bc5a361992c07b4bf4624f39720db04f5aa7de70cfa5"}
Sep 16 19:26:17.548252 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:26:17.548218    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" event={"ID":"8587aaaa-6e76-4930-afdc-8c28861106bb","Type":"ContainerStarted","Data":"9f98a3705be204703dbacd2f502ac21e7e515c4ed09401f79721ee192d1b29e0"}
Sep 16 19:26:17.548640 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:26:17.548306    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" event={"ID":"8587aaaa-6e76-4930-afdc-8c28861106bb","Type":"ContainerStarted","Data":"74a43dc0d63926a3f4d9800b6623e985925f62cefa485345feea9007ad63eb60"}
Sep 16 19:26:17.575726 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:26:17.575677    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" podStartSLOduration=2.646734754 podStartE2EDuration="37.575661035s" totalImagesPullingTime="33.744068826s" totalInitContainerRuntime="1.184857455s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:26:17.572624298 +0000 UTC m=+2815.932142294" watchObservedRunningTime="2026-09-16 19:26:17.575661035 +0000 UTC m=+2815.935179064"
Sep 16 19:26:17.969233 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:26:17.969149    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:26:20.969327 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:26:20.969288    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:26:20.971039 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:26:20.971002    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 19:26:30.969069 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:26:30.969021    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 19:26:40.969079 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:26:40.969035    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 19:26:50.969753 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:26:50.969704    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 19:27:00.968840 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:27:00.968789    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 19:27:10.969290 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:27:10.969248    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 19:27:20.969439 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:27:20.969395    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 19:27:30.969939 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:27:30.969887    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 19:27:40.969942 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:27:40.969889    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 19:27:50.969864 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:27:50.969811    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 19:28:00.969498 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:00.969438    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 19:28:10.979480 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:10.979444    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:28:10.987886 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:10.987850    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:28:15.924533 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:15.924489    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"]
Sep 16 19:28:15.925037 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:15.924872    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="modelcar" containerID="cri-o://9f98a3705be204703dbacd2f502ac21e7e515c4ed09401f79721ee192d1b29e0" gracePeriod=60
Sep 16 19:28:16.256161 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:16.256129    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk_8587aaaa-6e76-4930-afdc-8c28861106bb/modelcar/0.log"
Sep 16 19:28:16.256877 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:16.256850    2565 generic.go:356] "Generic (PLEG): container finished" podID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerID="9f98a3705be204703dbacd2f502ac21e7e515c4ed09401f79721ee192d1b29e0" exitCode=143
Sep 16 19:28:16.256985 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:16.256927    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" event={"ID":"8587aaaa-6e76-4930-afdc-8c28861106bb","Type":"ContainerDied","Data":"9f98a3705be204703dbacd2f502ac21e7e515c4ed09401f79721ee192d1b29e0"}
Sep 16 19:28:30.949264 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:30.949192    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="main" containerID="cri-o://74a43dc0d63926a3f4d9800b6623e985925f62cefa485345feea9007ad63eb60" gracePeriod=45
Sep 16 19:28:31.823049 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:31.823022    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk_8587aaaa-6e76-4930-afdc-8c28861106bb/modelcar/0.log"
Sep 16 19:28:31.825863 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:31.825840    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:28:31.958029 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:31.957984    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-home\") pod \"8587aaaa-6e76-4930-afdc-8c28861106bb\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") "
Sep 16 19:28:31.958029 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:31.958037    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8587aaaa-6e76-4930-afdc-8c28861106bb-tls-certs\") pod \"8587aaaa-6e76-4930-afdc-8c28861106bb\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") "
Sep 16 19:28:31.958644 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:31.958071    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-kserve-provision-location\") pod \"8587aaaa-6e76-4930-afdc-8c28861106bb\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") "
Sep 16 19:28:31.958644 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:31.958089    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-tmp-dir\") pod \"8587aaaa-6e76-4930-afdc-8c28861106bb\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") "
Sep 16 19:28:31.958644 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:31.958125    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9r9q\" (UniqueName: \"kubernetes.io/projected/8587aaaa-6e76-4930-afdc-8c28861106bb-kube-api-access-d9r9q\") pod \"8587aaaa-6e76-4930-afdc-8c28861106bb\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") "
Sep 16 19:28:31.958644 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:31.958216    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-model-cache\") pod \"8587aaaa-6e76-4930-afdc-8c28861106bb\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") "
Sep 16 19:28:31.958644 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:31.958238    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-dshm\") pod \"8587aaaa-6e76-4930-afdc-8c28861106bb\" (UID: \"8587aaaa-6e76-4930-afdc-8c28861106bb\") "
Sep 16 19:28:31.958644 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:31.958506    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8587aaaa-6e76-4930-afdc-8c28861106bb" (UID: "8587aaaa-6e76-4930-afdc-8c28861106bb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:31.958644 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:31.958521    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "8587aaaa-6e76-4930-afdc-8c28861106bb" (UID: "8587aaaa-6e76-4930-afdc-8c28861106bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:31.958976 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:31.958817    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-home" (OuterVolumeSpecName: "home") pod "8587aaaa-6e76-4930-afdc-8c28861106bb" (UID: "8587aaaa-6e76-4930-afdc-8c28861106bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:31.960598 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:31.960572    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-dshm" (OuterVolumeSpecName: "dshm") pod "8587aaaa-6e76-4930-afdc-8c28861106bb" (UID: "8587aaaa-6e76-4930-afdc-8c28861106bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:31.960776 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:31.960755    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8587aaaa-6e76-4930-afdc-8c28861106bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8587aaaa-6e76-4930-afdc-8c28861106bb" (UID: "8587aaaa-6e76-4930-afdc-8c28861106bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:28:31.960836 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:31.960771    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8587aaaa-6e76-4930-afdc-8c28861106bb-kube-api-access-d9r9q" (OuterVolumeSpecName: "kube-api-access-d9r9q") pod "8587aaaa-6e76-4930-afdc-8c28861106bb" (UID: "8587aaaa-6e76-4930-afdc-8c28861106bb"). InnerVolumeSpecName "kube-api-access-d9r9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:28:31.969845 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:31.969803    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8587aaaa-6e76-4930-afdc-8c28861106bb" (UID: "8587aaaa-6e76-4930-afdc-8c28861106bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:32.059675 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.059586    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9r9q\" (UniqueName: \"kubernetes.io/projected/8587aaaa-6e76-4930-afdc-8c28861106bb-kube-api-access-d9r9q\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:28:32.059675 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.059623    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:28:32.059675 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.059633    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:28:32.059675 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.059641    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:28:32.059675 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.059650    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8587aaaa-6e76-4930-afdc-8c28861106bb-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:28:32.059675 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.059658    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-kserve-provision-location\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:28:32.059675 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.059667    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8587aaaa-6e76-4930-afdc-8c28861106bb-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:28:32.349048 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.348957    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk_8587aaaa-6e76-4930-afdc-8c28861106bb/modelcar/0.log"
Sep 16 19:28:32.349764 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.349741    2565 generic.go:356] "Generic (PLEG): container finished" podID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerID="74a43dc0d63926a3f4d9800b6623e985925f62cefa485345feea9007ad63eb60" exitCode=0
Sep 16 19:28:32.349854 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.349811    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"
Sep 16 19:28:32.349854 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.349817    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" event={"ID":"8587aaaa-6e76-4930-afdc-8c28861106bb","Type":"ContainerDied","Data":"74a43dc0d63926a3f4d9800b6623e985925f62cefa485345feea9007ad63eb60"}
Sep 16 19:28:32.349929 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.349871    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk" event={"ID":"8587aaaa-6e76-4930-afdc-8c28861106bb","Type":"ContainerDied","Data":"05aca42b7443c86af1ce16172d198ebb542fbd9bfd3f28a6cb99f436ab7fd0ea"}
Sep 16 19:28:32.349929 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.349898    2565 scope.go:117] "RemoveContainer" containerID="9f98a3705be204703dbacd2f502ac21e7e515c4ed09401f79721ee192d1b29e0"
Sep 16 19:28:32.359241 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.359219    2565 scope.go:117] "RemoveContainer" containerID="74a43dc0d63926a3f4d9800b6623e985925f62cefa485345feea9007ad63eb60"
Sep 16 19:28:32.368632 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.368607    2565 scope.go:117] "RemoveContainer" containerID="7f20aed0a13664a54af9bc5a361992c07b4bf4624f39720db04f5aa7de70cfa5"
Sep 16 19:28:32.371830 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.371801    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"]
Sep 16 19:28:32.375733 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.375706    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-s9dgk"]
Sep 16 19:28:32.378757 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.378734    2565 scope.go:117] "RemoveContainer" containerID="9f98a3705be204703dbacd2f502ac21e7e515c4ed09401f79721ee192d1b29e0"
Sep 16 19:28:32.379079 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:28:32.379058    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f98a3705be204703dbacd2f502ac21e7e515c4ed09401f79721ee192d1b29e0\": container with ID starting with 9f98a3705be204703dbacd2f502ac21e7e515c4ed09401f79721ee192d1b29e0 not found: ID does not exist" containerID="9f98a3705be204703dbacd2f502ac21e7e515c4ed09401f79721ee192d1b29e0"
Sep 16 19:28:32.379163 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.379088    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f98a3705be204703dbacd2f502ac21e7e515c4ed09401f79721ee192d1b29e0"} err="failed to get container status \"9f98a3705be204703dbacd2f502ac21e7e515c4ed09401f79721ee192d1b29e0\": rpc error: code = NotFound desc = could not find container \"9f98a3705be204703dbacd2f502ac21e7e515c4ed09401f79721ee192d1b29e0\": container with ID starting with 9f98a3705be204703dbacd2f502ac21e7e515c4ed09401f79721ee192d1b29e0 not found: ID does not exist"
Sep 16 19:28:32.379163 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.379110    2565 scope.go:117] "RemoveContainer" containerID="74a43dc0d63926a3f4d9800b6623e985925f62cefa485345feea9007ad63eb60"
Sep 16 19:28:32.379373 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:28:32.379358    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a43dc0d63926a3f4d9800b6623e985925f62cefa485345feea9007ad63eb60\": container with ID starting with 74a43dc0d63926a3f4d9800b6623e985925f62cefa485345feea9007ad63eb60 not found: ID does not exist" containerID="74a43dc0d63926a3f4d9800b6623e985925f62cefa485345feea9007ad63eb60"
Sep 16 19:28:32.379430 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.379375    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a43dc0d63926a3f4d9800b6623e985925f62cefa485345feea9007ad63eb60"} err="failed to get container status \"74a43dc0d63926a3f4d9800b6623e985925f62cefa485345feea9007ad63eb60\": rpc error: code = NotFound desc = could not find container \"74a43dc0d63926a3f4d9800b6623e985925f62cefa485345feea9007ad63eb60\": container with ID starting with 74a43dc0d63926a3f4d9800b6623e985925f62cefa485345feea9007ad63eb60 not found: ID does not exist"
Sep 16 19:28:32.379430 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.379388    2565 scope.go:117] "RemoveContainer" containerID="7f20aed0a13664a54af9bc5a361992c07b4bf4624f39720db04f5aa7de70cfa5"
Sep 16 19:28:32.379604 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:28:32.379585    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f20aed0a13664a54af9bc5a361992c07b4bf4624f39720db04f5aa7de70cfa5\": container with ID starting with 7f20aed0a13664a54af9bc5a361992c07b4bf4624f39720db04f5aa7de70cfa5 not found: ID does not exist" containerID="7f20aed0a13664a54af9bc5a361992c07b4bf4624f39720db04f5aa7de70cfa5"
Sep 16 19:28:32.379647 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:32.379610    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f20aed0a13664a54af9bc5a361992c07b4bf4624f39720db04f5aa7de70cfa5"} err="failed to get container status \"7f20aed0a13664a54af9bc5a361992c07b4bf4624f39720db04f5aa7de70cfa5\": rpc error: code = NotFound desc = could not find container \"7f20aed0a13664a54af9bc5a361992c07b4bf4624f39720db04f5aa7de70cfa5\": container with ID starting with 7f20aed0a13664a54af9bc5a361992c07b4bf4624f39720db04f5aa7de70cfa5 not found: ID does not exist"
Sep 16 19:28:34.207753 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:28:34.207716    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" path="/var/lib/kubelet/pods/8587aaaa-6e76-4930-afdc-8c28861106bb/volumes"
Sep 16 19:32:08.484218 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.484177    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"]
Sep 16 19:32:08.484954 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.484791    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="modelcar-init"
Sep 16 19:32:08.484954 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.484815    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="modelcar-init"
Sep 16 19:32:08.484954 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.484830    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="modelcar"
Sep 16 19:32:08.484954 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.484840    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="modelcar"
Sep 16 19:32:08.484954 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.484868    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="main"
Sep 16 19:32:08.484954 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.484877    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="main"
Sep 16 19:32:08.485317 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.484984    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="main"
Sep 16 19:32:08.485317 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.484995    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8587aaaa-6e76-4930-afdc-8c28861106bb" containerName="modelcar"
Sep 16 19:32:08.488666 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.488636    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.491493 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.491457    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-cs6wm\""
Sep 16 19:32:08.491644 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.491470    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:32:08.491644 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.491589    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 19:32:08.491854 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.491832    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 19:32:08.504773 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.504729    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"]
Sep 16 19:32:08.598792 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.598748    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-envoy\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.599018 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.598828    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-workload-certs\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.599018 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.598869    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.599018 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.598927    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-data\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.599018 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.598971    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwfnz\" (UniqueName: \"kubernetes.io/projected/2dd93fbb-0849-4e02-b8e2-91af0a42443c-kube-api-access-pwfnz\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.599018 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.599007    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-podinfo\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.599282 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.599061    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-workload-socket\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.599282 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.599121    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-credential-socket\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.599282 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.599164    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-token\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.700339 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.700285    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-credential-socket\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.700339 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.700376    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-token\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.700651 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.700412    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-envoy\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.700651 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.700437    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-workload-certs\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.700651 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.700460    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.700651 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.700490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-data\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.700651 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.700516    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwfnz\" (UniqueName: \"kubernetes.io/projected/2dd93fbb-0849-4e02-b8e2-91af0a42443c-kube-api-access-pwfnz\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.700924 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.700804    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-podinfo\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.700924 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.700846    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-workload-certs\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.700924 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.700875    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-workload-socket\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.701063 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.700935    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-credential-socket\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.701063 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.700939    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-data\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.701205 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.701187    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-workload-socket\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.701385 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.701327    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.702994 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.702969    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-envoy\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.703289 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.703266    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-podinfo\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.709255 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.709220    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-token\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.709548 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.709521    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwfnz\" (UniqueName: \"kubernetes.io/projected/2dd93fbb-0849-4e02-b8e2-91af0a42443c-kube-api-access-pwfnz\") pod \"router-gateway-1-openshift-default-65f799fdbd-q984l\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:08.801173 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:08.801055    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:09.091945 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:09.091911    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"]
Sep 16 19:32:09.094300 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:32:09.094253    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd93fbb_0849_4e02_b8e2_91af0a42443c.slice/crio-352aa2169ef848a74481bdb79f23cc37d95a2411512a1086f3a3ffb0e390a364 WatchSource:0}: Error finding container 352aa2169ef848a74481bdb79f23cc37d95a2411512a1086f3a3ffb0e390a364: Status 404 returned error can't find the container with id 352aa2169ef848a74481bdb79f23cc37d95a2411512a1086f3a3ffb0e390a364
Sep 16 19:32:09.096534 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:09.096492    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:32:09.096642 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:09.096568    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:32:09.096642 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:09.096602    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:32:09.553730 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:09.553694    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l" event={"ID":"2dd93fbb-0849-4e02-b8e2-91af0a42443c","Type":"ContainerStarted","Data":"054515f63115c89e25e1adac72648246af5ec84ca08e415a98e9d050e5465348"}
Sep 16 19:32:09.554135 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:09.553751    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l" event={"ID":"2dd93fbb-0849-4e02-b8e2-91af0a42443c","Type":"ContainerStarted","Data":"352aa2169ef848a74481bdb79f23cc37d95a2411512a1086f3a3ffb0e390a364"}
Sep 16 19:32:09.575549 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:09.575479    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l" podStartSLOduration=1.575462576 podStartE2EDuration="1.575462576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:32:09.573512037 +0000 UTC m=+3167.933030031" watchObservedRunningTime="2026-09-16 19:32:09.575462576 +0000 UTC m=+3167.934980571"
Sep 16 19:32:09.801369 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:09.801302    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:10.804413 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:10.804335    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l" podUID="2dd93fbb-0849-4e02-b8e2-91af0a42443c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 16 19:32:11.802375 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:11.801900    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l" podUID="2dd93fbb-0849-4e02-b8e2-91af0a42443c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 16 19:32:12.801885 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:12.801839    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l" podUID="2dd93fbb-0849-4e02-b8e2-91af0a42443c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 16 19:32:13.802130 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:13.802074    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l" podUID="2dd93fbb-0849-4e02-b8e2-91af0a42443c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 16 19:32:14.801935 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:14.801878    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l" podUID="2dd93fbb-0849-4e02-b8e2-91af0a42443c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 16 19:32:15.806383 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:15.806332    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:15.806898 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:15.806746    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:32:15.807649 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:32:15.807630    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:35:21.222898 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:21.222843    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"]
Sep 16 19:35:21.223508 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:21.223066    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l" podUID="2dd93fbb-0849-4e02-b8e2-91af0a42443c" containerName="istio-proxy" containerID="cri-o://054515f63115c89e25e1adac72648246af5ec84ca08e415a98e9d050e5465348" gracePeriod=30
Sep 16 19:35:26.470277 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.470246    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:35:26.620748 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.620696    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwfnz\" (UniqueName: \"kubernetes.io/projected/2dd93fbb-0849-4e02-b8e2-91af0a42443c-kube-api-access-pwfnz\") pod \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") "
Sep 16 19:35:26.620960 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.620757    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-podinfo\") pod \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") "
Sep 16 19:35:26.620960 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.620792    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-credential-socket\") pod \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") "
Sep 16 19:35:26.620960 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.620818    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-envoy\") pod \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") "
Sep 16 19:35:26.620960 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.620860    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istiod-ca-cert\") pod \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") "
Sep 16 19:35:26.620960 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.620921    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-data\") pod \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") "
Sep 16 19:35:26.620960 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.620947    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-workload-socket\") pod \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") "
Sep 16 19:35:26.621378 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.621005    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-token\") pod \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") "
Sep 16 19:35:26.621378 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.621043    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-workload-certs\") pod \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\" (UID: \"2dd93fbb-0849-4e02-b8e2-91af0a42443c\") "
Sep 16 19:35:26.621378 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.621155    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2dd93fbb-0849-4e02-b8e2-91af0a42443c" (UID: "2dd93fbb-0849-4e02-b8e2-91af0a42443c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:26.621378 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.621191    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-data" (OuterVolumeSpecName: "istio-data") pod "2dd93fbb-0849-4e02-b8e2-91af0a42443c" (UID: "2dd93fbb-0849-4e02-b8e2-91af0a42443c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:26.621378 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.621260    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2dd93fbb-0849-4e02-b8e2-91af0a42443c" (UID: "2dd93fbb-0849-4e02-b8e2-91af0a42443c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:35:26.621645 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.621454    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2dd93fbb-0849-4e02-b8e2-91af0a42443c" (UID: "2dd93fbb-0849-4e02-b8e2-91af0a42443c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:26.621645 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.621463    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istiod-ca-cert\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:35:26.621645 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.621529    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-data\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:35:26.621645 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.621546    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-credential-socket\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:35:26.621845 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.621722    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2dd93fbb-0849-4e02-b8e2-91af0a42443c" (UID: "2dd93fbb-0849-4e02-b8e2-91af0a42443c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:26.623257 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.623225    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2dd93fbb-0849-4e02-b8e2-91af0a42443c" (UID: "2dd93fbb-0849-4e02-b8e2-91af0a42443c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:35:26.623432 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.623268    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2dd93fbb-0849-4e02-b8e2-91af0a42443c" (UID: "2dd93fbb-0849-4e02-b8e2-91af0a42443c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:26.623432 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.623331    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd93fbb-0849-4e02-b8e2-91af0a42443c-kube-api-access-pwfnz" (OuterVolumeSpecName: "kube-api-access-pwfnz") pod "2dd93fbb-0849-4e02-b8e2-91af0a42443c" (UID: "2dd93fbb-0849-4e02-b8e2-91af0a42443c"). InnerVolumeSpecName "kube-api-access-pwfnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:26.623432 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.623375    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-token" (OuterVolumeSpecName: "istio-token") pod "2dd93fbb-0849-4e02-b8e2-91af0a42443c" (UID: "2dd93fbb-0849-4e02-b8e2-91af0a42443c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:26.678238 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.678202    2565 generic.go:356] "Generic (PLEG): container finished" podID="2dd93fbb-0849-4e02-b8e2-91af0a42443c" containerID="054515f63115c89e25e1adac72648246af5ec84ca08e415a98e9d050e5465348" exitCode=0
Sep 16 19:35:26.678451 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.678251    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"
Sep 16 19:35:26.678451 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.678291    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l" event={"ID":"2dd93fbb-0849-4e02-b8e2-91af0a42443c","Type":"ContainerDied","Data":"054515f63115c89e25e1adac72648246af5ec84ca08e415a98e9d050e5465348"}
Sep 16 19:35:26.678451 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.678375    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l" event={"ID":"2dd93fbb-0849-4e02-b8e2-91af0a42443c","Type":"ContainerDied","Data":"352aa2169ef848a74481bdb79f23cc37d95a2411512a1086f3a3ffb0e390a364"}
Sep 16 19:35:26.678451 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.678402    2565 scope.go:117] "RemoveContainer" containerID="054515f63115c89e25e1adac72648246af5ec84ca08e415a98e9d050e5465348"
Sep 16 19:35:26.688966 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.688939    2565 scope.go:117] "RemoveContainer" containerID="054515f63115c89e25e1adac72648246af5ec84ca08e415a98e9d050e5465348"
Sep 16 19:35:26.689324 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:35:26.689301    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"054515f63115c89e25e1adac72648246af5ec84ca08e415a98e9d050e5465348\": container with ID starting with 054515f63115c89e25e1adac72648246af5ec84ca08e415a98e9d050e5465348 not found: ID does not exist" containerID="054515f63115c89e25e1adac72648246af5ec84ca08e415a98e9d050e5465348"
Sep 16 19:35:26.689487 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.689338    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"054515f63115c89e25e1adac72648246af5ec84ca08e415a98e9d050e5465348"} err="failed to get container status \"054515f63115c89e25e1adac72648246af5ec84ca08e415a98e9d050e5465348\": rpc error: code = NotFound desc = could not find container \"054515f63115c89e25e1adac72648246af5ec84ca08e415a98e9d050e5465348\": container with ID starting with 054515f63115c89e25e1adac72648246af5ec84ca08e415a98e9d050e5465348 not found: ID does not exist"
Sep 16 19:35:26.705294 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.705253    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"]
Sep 16 19:35:26.711455 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.711420    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-65f799fdbd-q984l"]
Sep 16 19:35:26.722082 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.722056    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-token\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:35:26.722082 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.722079    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-workload-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:35:26.722198 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.722089    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwfnz\" (UniqueName: \"kubernetes.io/projected/2dd93fbb-0849-4e02-b8e2-91af0a42443c-kube-api-access-pwfnz\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:35:26.722198 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.722098    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-podinfo\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:35:26.722198 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.722108    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-istio-envoy\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:35:26.722198 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:26.722116    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd93fbb-0849-4e02-b8e2-91af0a42443c-workload-socket\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:35:28.208324 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:28.208286    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dd93fbb-0849-4e02-b8e2-91af0a42443c" path="/var/lib/kubelet/pods/2dd93fbb-0849-4e02-b8e2-91af0a42443c/volumes"
Sep 16 19:35:36.617584 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:36.617541    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j"]
Sep 16 19:35:36.618192 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:36.618173    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd93fbb-0849-4e02-b8e2-91af0a42443c" containerName="istio-proxy"
Sep 16 19:35:36.618249 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:36.618196    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd93fbb-0849-4e02-b8e2-91af0a42443c" containerName="istio-proxy"
Sep 16 19:35:36.618311 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:36.618299    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd93fbb-0849-4e02-b8e2-91af0a42443c" containerName="istio-proxy"
Sep 16 19:35:36.621907 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:36.621875    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j"
Sep 16 19:35:36.625404 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:36.625370    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 19:35:36.626468 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:36.626441    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-pggx2\""
Sep 16 19:35:36.626610 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:36.626498    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 19:35:36.626610 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:36.626569    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 19:35:36.626610 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:36.626509    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-72bgx\""
Sep 16 19:35:36.640812 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:36.640775    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j"]
Sep 16 19:35:36.814576 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:36.814527    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67dbdaf1-1f24-4573-aadd-85b88f6d02dc-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j\" (UID: \"67dbdaf1-1f24-4573-aadd-85b88f6d02dc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j"
Sep 16 19:35:36.814576 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:36.814578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdlk6\" (UniqueName: \"kubernetes.io/projected/67dbdaf1-1f24-4573-aadd-85b88f6d02dc-kube-api-access-wdlk6\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j\" (UID: \"67dbdaf1-1f24-4573-aadd-85b88f6d02dc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j"
Sep 16 19:35:36.915159 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:36.915055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdlk6\" (UniqueName: \"kubernetes.io/projected/67dbdaf1-1f24-4573-aadd-85b88f6d02dc-kube-api-access-wdlk6\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j\" (UID: \"67dbdaf1-1f24-4573-aadd-85b88f6d02dc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j"
Sep 16 19:35:36.915376 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:36.915164    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67dbdaf1-1f24-4573-aadd-85b88f6d02dc-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j\" (UID: \"67dbdaf1-1f24-4573-aadd-85b88f6d02dc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j"
Sep 16 19:35:36.917972 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:36.917941    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67dbdaf1-1f24-4573-aadd-85b88f6d02dc-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j\" (UID: \"67dbdaf1-1f24-4573-aadd-85b88f6d02dc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j"
Sep 16 19:35:36.935166 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:36.935128    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdlk6\" (UniqueName: \"kubernetes.io/projected/67dbdaf1-1f24-4573-aadd-85b88f6d02dc-kube-api-access-wdlk6\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j\" (UID: \"67dbdaf1-1f24-4573-aadd-85b88f6d02dc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j"
Sep 16 19:35:37.233266 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:37.233218    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j"
Sep 16 19:35:37.526334 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:37.526306    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j"]
Sep 16 19:35:37.528850 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:35:37.528819    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67dbdaf1_1f24_4573_aadd_85b88f6d02dc.slice/crio-ac7f3e939b6b0f6ede06ecd9cbcb1282008a7c1ce722c47824a01edf4fa67a1e WatchSource:0}: Error finding container ac7f3e939b6b0f6ede06ecd9cbcb1282008a7c1ce722c47824a01edf4fa67a1e: Status 404 returned error can't find the container with id ac7f3e939b6b0f6ede06ecd9cbcb1282008a7c1ce722c47824a01edf4fa67a1e
Sep 16 19:35:37.743158 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:37.743110    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j" event={"ID":"67dbdaf1-1f24-4573-aadd-85b88f6d02dc","Type":"ContainerStarted","Data":"e8beda0c1b947bd6393b9edc8ed82d5ee44ed42f377db4febe1ecb3e9fdaa866"}
Sep 16 19:35:37.743693 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:37.743213    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j"
Sep 16 19:35:37.743693 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:37.743238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j" event={"ID":"67dbdaf1-1f24-4573-aadd-85b88f6d02dc","Type":"ContainerStarted","Data":"ac7f3e939b6b0f6ede06ecd9cbcb1282008a7c1ce722c47824a01edf4fa67a1e"}
Sep 16 19:35:37.764652 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:35:37.764585    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j" podStartSLOduration=1.764566984 podStartE2EDuration="1.764566984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:37.76296555 +0000 UTC m=+3376.122483569" watchObservedRunningTime="2026-09-16 19:35:37.764566984 +0000 UTC m=+3376.124084979"
Sep 16 19:36:08.752682 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:36:08.752646    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j"
Sep 16 19:38:25.952248 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:38:25.952206    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j"]
Sep 16 19:38:28.752334 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:38:28.752285    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j" podUID="67dbdaf1-1f24-4573-aadd-85b88f6d02dc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:38:38.752754 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:38:38.752705    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j" podUID="67dbdaf1-1f24-4573-aadd-85b88f6d02dc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:38:40.972542 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:38:40.972492    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j" podUID="67dbdaf1-1f24-4573-aadd-85b88f6d02dc" containerName="main" containerID="cri-o://e8beda0c1b947bd6393b9edc8ed82d5ee44ed42f377db4febe1ecb3e9fdaa866" gracePeriod=45
Sep 16 19:38:48.752203 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:38:48.752159    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j" podUID="67dbdaf1-1f24-4573-aadd-85b88f6d02dc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:38:48.752627 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:38:48.752293    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j"
Sep 16 19:38:58.752327 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:38:58.752285    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j" podUID="67dbdaf1-1f24-4573-aadd-85b88f6d02dc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:39:08.752590 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:39:08.752546    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j" podUID="67dbdaf1-1f24-4573-aadd-85b88f6d02dc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:39:11.240459 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:39:11.240379    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j"
Sep 16 19:39:11.261221 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:39:11.261183    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdlk6\" (UniqueName: \"kubernetes.io/projected/67dbdaf1-1f24-4573-aadd-85b88f6d02dc-kube-api-access-wdlk6\") pod \"67dbdaf1-1f24-4573-aadd-85b88f6d02dc\" (UID: \"67dbdaf1-1f24-4573-aadd-85b88f6d02dc\") "
Sep 16 19:39:11.261444 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:39:11.261261    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67dbdaf1-1f24-4573-aadd-85b88f6d02dc-tls-certs\") pod \"67dbdaf1-1f24-4573-aadd-85b88f6d02dc\" (UID: \"67dbdaf1-1f24-4573-aadd-85b88f6d02dc\") "
Sep 16 19:39:11.263801 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:39:11.263758    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67dbdaf1-1f24-4573-aadd-85b88f6d02dc-kube-api-access-wdlk6" (OuterVolumeSpecName: "kube-api-access-wdlk6") pod "67dbdaf1-1f24-4573-aadd-85b88f6d02dc" (UID: "67dbdaf1-1f24-4573-aadd-85b88f6d02dc"). InnerVolumeSpecName "kube-api-access-wdlk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:39:11.263958 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:39:11.263835    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67dbdaf1-1f24-4573-aadd-85b88f6d02dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67dbdaf1-1f24-4573-aadd-85b88f6d02dc" (UID: "67dbdaf1-1f24-4573-aadd-85b88f6d02dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:39:11.362745 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:39:11.362693    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67dbdaf1-1f24-4573-aadd-85b88f6d02dc-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:39:11.362745 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:39:11.362734    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdlk6\" (UniqueName: \"kubernetes.io/projected/67dbdaf1-1f24-4573-aadd-85b88f6d02dc-kube-api-access-wdlk6\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:39:11.920109 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:39:11.920069    2565 generic.go:356] "Generic (PLEG): container finished" podID="67dbdaf1-1f24-4573-aadd-85b88f6d02dc" containerID="e8beda0c1b947bd6393b9edc8ed82d5ee44ed42f377db4febe1ecb3e9fdaa866" exitCode=0
Sep 16 19:39:11.920319 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:39:11.920112    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j"
Sep 16 19:39:11.920319 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:39:11.920151    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j" event={"ID":"67dbdaf1-1f24-4573-aadd-85b88f6d02dc","Type":"ContainerDied","Data":"e8beda0c1b947bd6393b9edc8ed82d5ee44ed42f377db4febe1ecb3e9fdaa866"}
Sep 16 19:39:11.920319 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:39:11.920203    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j" event={"ID":"67dbdaf1-1f24-4573-aadd-85b88f6d02dc","Type":"ContainerDied","Data":"ac7f3e939b6b0f6ede06ecd9cbcb1282008a7c1ce722c47824a01edf4fa67a1e"}
Sep 16 19:39:11.920319 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:39:11.920243    2565 scope.go:117] "RemoveContainer" containerID="e8beda0c1b947bd6393b9edc8ed82d5ee44ed42f377db4febe1ecb3e9fdaa866"
Sep 16 19:39:11.930390 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:39:11.930360    2565 scope.go:117] "RemoveContainer" containerID="e8beda0c1b947bd6393b9edc8ed82d5ee44ed42f377db4febe1ecb3e9fdaa866"
Sep 16 19:39:11.930704 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:39:11.930682    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8beda0c1b947bd6393b9edc8ed82d5ee44ed42f377db4febe1ecb3e9fdaa866\": container with ID starting with e8beda0c1b947bd6393b9edc8ed82d5ee44ed42f377db4febe1ecb3e9fdaa866 not found: ID does not exist" containerID="e8beda0c1b947bd6393b9edc8ed82d5ee44ed42f377db4febe1ecb3e9fdaa866"
Sep 16 19:39:11.930749 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:39:11.930714    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8beda0c1b947bd6393b9edc8ed82d5ee44ed42f377db4febe1ecb3e9fdaa866"} err="failed to get container status \"e8beda0c1b947bd6393b9edc8ed82d5ee44ed42f377db4febe1ecb3e9fdaa866\": rpc error: code = NotFound desc = could not find container \"e8beda0c1b947bd6393b9edc8ed82d5ee44ed42f377db4febe1ecb3e9fdaa866\": container with ID starting with e8beda0c1b947bd6393b9edc8ed82d5ee44ed42f377db4febe1ecb3e9fdaa866 not found: ID does not exist"
Sep 16 19:39:11.942911 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:39:11.942877    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j"]
Sep 16 19:39:11.946755 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:39:11.946713    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegd94j"]
Sep 16 19:39:12.207516 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:39:12.207413    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67dbdaf1-1f24-4573-aadd-85b88f6d02dc" path="/var/lib/kubelet/pods/67dbdaf1-1f24-4573-aadd-85b88f6d02dc/volumes"
Sep 16 19:49:05.292250 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.292209    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"]
Sep 16 19:49:05.294743 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.292635    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67dbdaf1-1f24-4573-aadd-85b88f6d02dc" containerName="main"
Sep 16 19:49:05.294743 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.292649    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="67dbdaf1-1f24-4573-aadd-85b88f6d02dc" containerName="main"
Sep 16 19:49:05.294743 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.292734    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="67dbdaf1-1f24-4573-aadd-85b88f6d02dc" containerName="main"
Sep 16 19:49:05.295859 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.295838    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.304649 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.304625    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-lb66q\""
Sep 16 19:49:05.305410 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.305387    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 19:49:05.305490 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.305387    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 19:49:05.305490 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.305461    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 19:49:05.319537 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.319499    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"]
Sep 16 19:49:05.408256 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.408208    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-fv68x\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.408501 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.408275    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-fv68x\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.408501 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.408301    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-fv68x\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.408501 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.408330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-fv68x\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.408501 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.408391    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-fv68x\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.408501 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.408421    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzx5q\" (UniqueName: \"kubernetes.io/projected/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-kube-api-access-nzx5q\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-fv68x\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.509871 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.509822    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-fv68x\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.509871 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.509876    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-fv68x\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.510130 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.509900    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzx5q\" (UniqueName: \"kubernetes.io/projected/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-kube-api-access-nzx5q\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-fv68x\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.510130 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.509965    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-fv68x\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.510130 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.510002    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-fv68x\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.510130 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.510020    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-fv68x\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.510398 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.510377    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-fv68x\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.510460 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.510381    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-fv68x\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.510460 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.510413    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-fv68x\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.512442 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.512419    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-fv68x\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.512605 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.512588    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-fv68x\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.521234 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.521205    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzx5q\" (UniqueName: \"kubernetes.io/projected/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-kube-api-access-nzx5q\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-fv68x\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.607159 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.607067    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:05.899627 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:05.899578    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"]
Sep 16 19:49:05.902171 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:49:05.902137    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ccb38c_2a05_4d3f_8f6f_b8b0d158b3cf.slice/crio-9b738ef693d1d6f42debd2af642f37b3014b42693c33ace4cf2049cc85022ff4 WatchSource:0}: Error finding container 9b738ef693d1d6f42debd2af642f37b3014b42693c33ace4cf2049cc85022ff4: Status 404 returned error can't find the container with id 9b738ef693d1d6f42debd2af642f37b3014b42693c33ace4cf2049cc85022ff4
Sep 16 19:49:06.278180 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:06.278132    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x" event={"ID":"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf","Type":"ContainerStarted","Data":"54a5f0d436e3d76197323416a2ae77e91cacba58b815093f88be1b7fdc3f6861"}
Sep 16 19:49:06.278393 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:06.278207    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x" event={"ID":"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf","Type":"ContainerStarted","Data":"9b738ef693d1d6f42debd2af642f37b3014b42693c33ace4cf2049cc85022ff4"}
Sep 16 19:49:06.308999 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:06.308934    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x" podStartSLOduration=1.3089177379999999 podStartE2EDuration="1.308917738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:49:06.305549699 +0000 UTC m=+4184.665067695" watchObservedRunningTime="2026-09-16 19:49:06.308917738 +0000 UTC m=+4184.668435733"
Sep 16 19:49:06.608038 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:06.607927    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:10.790650 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:10.790605    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"]
Sep 16 19:49:10.801197 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:49:10.801095    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:49:10.801197 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:49:10.801197 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 19:49:10.801197 ip-10-0-130-204 kubenswrapper[2565]:  > containerID="54a5f0d436e3d76197323416a2ae77e91cacba58b815093f88be1b7fdc3f6861" cmd=["/bin/sleep","15"]
Sep 16 19:49:10.801550 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:10.801224    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x" message="" err=<
Sep 16 19:49:10.801550 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:49:10.801550 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 19:49:10.801550 ip-10-0-130-204 kubenswrapper[2565]:  >
Sep 16 19:49:10.801550 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:49:10.801243    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:49:10.801550 ip-10-0-130-204 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:49:10.801550 ip-10-0-130-204 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 16 19:49:10.801550 ip-10-0-130-204 kubenswrapper[2565]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x" podUID="22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf" containerName="main" containerID="cri-o://54a5f0d436e3d76197323416a2ae77e91cacba58b815093f88be1b7fdc3f6861"
Sep 16 19:49:10.801550 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:10.801317    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x" podUID="22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf" containerName="main" containerID="cri-o://54a5f0d436e3d76197323416a2ae77e91cacba58b815093f88be1b7fdc3f6861" gracePeriod=60
Sep 16 19:49:11.057601 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.057569    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:11.159168 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.159127    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-home\") pod \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") "
Sep 16 19:49:11.159381 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.159230    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-tls-certs\") pod \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") "
Sep 16 19:49:11.159381 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.159283    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-dshm\") pod \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") "
Sep 16 19:49:11.159381 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.159308    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-model-cache\") pod \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") "
Sep 16 19:49:11.159381 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.159372    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-tmp-dir\") pod \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") "
Sep 16 19:49:11.159625 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.159411    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzx5q\" (UniqueName: \"kubernetes.io/projected/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-kube-api-access-nzx5q\") pod \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\" (UID: \"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf\") "
Sep 16 19:49:11.159625 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.159453    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-home" (OuterVolumeSpecName: "home") pod "22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf" (UID: "22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:11.159625 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.159582    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf" (UID: "22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:11.159757 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.159608    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf" (UID: "22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:11.159757 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.159731    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-tmp-dir\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:49:11.159757 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.159751    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-home\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:49:11.159876 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.159769    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-model-cache\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:49:11.161727 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.161696    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-dshm" (OuterVolumeSpecName: "dshm") pod "22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf" (UID: "22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:11.161841 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.161739    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf" (UID: "22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:49:11.161841 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.161772    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-kube-api-access-nzx5q" (OuterVolumeSpecName: "kube-api-access-nzx5q") pod "22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf" (UID: "22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf"). InnerVolumeSpecName "kube-api-access-nzx5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:49:11.261218 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.261169    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-tls-certs\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:49:11.261218 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.261210    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-dshm\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:49:11.261218 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.261223    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzx5q\" (UniqueName: \"kubernetes.io/projected/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf-kube-api-access-nzx5q\") on node \"ip-10-0-130-204.ec2.internal\" DevicePath \"\""
Sep 16 19:49:11.307416 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.307321    2565 generic.go:356] "Generic (PLEG): container finished" podID="22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf" containerID="54a5f0d436e3d76197323416a2ae77e91cacba58b815093f88be1b7fdc3f6861" exitCode=0
Sep 16 19:49:11.307569 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.307408    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"
Sep 16 19:49:11.307569 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.307408    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x" event={"ID":"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf","Type":"ContainerDied","Data":"54a5f0d436e3d76197323416a2ae77e91cacba58b815093f88be1b7fdc3f6861"}
Sep 16 19:49:11.307569 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.307543    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x" event={"ID":"22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf","Type":"ContainerDied","Data":"9b738ef693d1d6f42debd2af642f37b3014b42693c33ace4cf2049cc85022ff4"}
Sep 16 19:49:11.307692 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.307576    2565 scope.go:117] "RemoveContainer" containerID="54a5f0d436e3d76197323416a2ae77e91cacba58b815093f88be1b7fdc3f6861"
Sep 16 19:49:11.319262 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.319234    2565 scope.go:117] "RemoveContainer" containerID="54a5f0d436e3d76197323416a2ae77e91cacba58b815093f88be1b7fdc3f6861"
Sep 16 19:49:11.319616 ip-10-0-130-204 kubenswrapper[2565]: E0916 19:49:11.319592    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54a5f0d436e3d76197323416a2ae77e91cacba58b815093f88be1b7fdc3f6861\": container with ID starting with 54a5f0d436e3d76197323416a2ae77e91cacba58b815093f88be1b7fdc3f6861 not found: ID does not exist" containerID="54a5f0d436e3d76197323416a2ae77e91cacba58b815093f88be1b7fdc3f6861"
Sep 16 19:49:11.319682 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.319629    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a5f0d436e3d76197323416a2ae77e91cacba58b815093f88be1b7fdc3f6861"} err="failed to get container status \"54a5f0d436e3d76197323416a2ae77e91cacba58b815093f88be1b7fdc3f6861\": rpc error: code = NotFound desc = could not find container \"54a5f0d436e3d76197323416a2ae77e91cacba58b815093f88be1b7fdc3f6861\": container with ID starting with 54a5f0d436e3d76197323416a2ae77e91cacba58b815093f88be1b7fdc3f6861 not found: ID does not exist"
Sep 16 19:49:11.344859 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.344815    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"]
Sep 16 19:49:11.357127 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:11.357087    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-fv68x"]
Sep 16 19:49:12.207561 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:49:12.207523    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf" path="/var/lib/kubelet/pods/22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf/volumes"
Sep 16 19:50:24.802796 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:24.802761    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qg4vd_e2adc2cd-afaf-4150-8963-7d2b524b47e4/discovery/0.log"
Sep 16 19:50:25.756161 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:25.756124    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qg4vd_e2adc2cd-afaf-4150-8963-7d2b524b47e4/discovery/0.log"
Sep 16 19:50:27.024875 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:27.024826    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-rkwxr_cd86797c-c8c8-4372-a44e-ca40a3c6a884/manager/0.log"
Sep 16 19:50:27.149956 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:27.149917    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-hf6t2_14eeb42f-1bd4-4b63-a462-487e6d1c7187/manager/0.log"
Sep 16 19:50:27.215127 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:27.215094    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tgvjm_f356ad9e-45a9-4a92-9352-68ede7f9c77b/manager/0.log"
Sep 16 19:50:33.339840 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:33.339797    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s865n/must-gather-7p59p"]
Sep 16 19:50:33.340466 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:33.340448    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf" containerName="main"
Sep 16 19:50:33.340562 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:33.340473    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf" containerName="main"
Sep 16 19:50:33.340630 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:33.340581    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="22ccb38c-2a05-4d3f-8f6f-b8b0d158b3cf" containerName="main"
Sep 16 19:50:33.344245 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:33.344211    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s865n/must-gather-7p59p"
Sep 16 19:50:33.346701 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:33.346664    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s865n\"/\"openshift-service-ca.crt\""
Sep 16 19:50:33.346701 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:33.346664    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s865n\"/\"kube-root-ca.crt\""
Sep 16 19:50:33.347587 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:33.347571    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s865n\"/\"default-dockercfg-b7pzr\""
Sep 16 19:50:33.354265 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:33.354236    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s865n/must-gather-7p59p"]
Sep 16 19:50:33.410734 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:33.410687    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpj8w\" (UniqueName: \"kubernetes.io/projected/7b12c0cf-66b3-48e1-b908-c7d1c6d47dd9-kube-api-access-zpj8w\") pod \"must-gather-7p59p\" (UID: \"7b12c0cf-66b3-48e1-b908-c7d1c6d47dd9\") " pod="openshift-must-gather-s865n/must-gather-7p59p"
Sep 16 19:50:33.410734 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:33.410739    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7b12c0cf-66b3-48e1-b908-c7d1c6d47dd9-must-gather-output\") pod \"must-gather-7p59p\" (UID: \"7b12c0cf-66b3-48e1-b908-c7d1c6d47dd9\") " pod="openshift-must-gather-s865n/must-gather-7p59p"
Sep 16 19:50:33.511632 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:33.511568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpj8w\" (UniqueName: \"kubernetes.io/projected/7b12c0cf-66b3-48e1-b908-c7d1c6d47dd9-kube-api-access-zpj8w\") pod \"must-gather-7p59p\" (UID: \"7b12c0cf-66b3-48e1-b908-c7d1c6d47dd9\") " pod="openshift-must-gather-s865n/must-gather-7p59p"
Sep 16 19:50:33.511632 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:33.511641    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7b12c0cf-66b3-48e1-b908-c7d1c6d47dd9-must-gather-output\") pod \"must-gather-7p59p\" (UID: \"7b12c0cf-66b3-48e1-b908-c7d1c6d47dd9\") " pod="openshift-must-gather-s865n/must-gather-7p59p"
Sep 16 19:50:33.511998 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:33.511970    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7b12c0cf-66b3-48e1-b908-c7d1c6d47dd9-must-gather-output\") pod \"must-gather-7p59p\" (UID: \"7b12c0cf-66b3-48e1-b908-c7d1c6d47dd9\") " pod="openshift-must-gather-s865n/must-gather-7p59p"
Sep 16 19:50:33.520394 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:33.520339    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpj8w\" (UniqueName: \"kubernetes.io/projected/7b12c0cf-66b3-48e1-b908-c7d1c6d47dd9-kube-api-access-zpj8w\") pod \"must-gather-7p59p\" (UID: \"7b12c0cf-66b3-48e1-b908-c7d1c6d47dd9\") " pod="openshift-must-gather-s865n/must-gather-7p59p"
Sep 16 19:50:33.655662 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:33.655563    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s865n/must-gather-7p59p"
Sep 16 19:50:33.936466 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:33.936372    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s865n/must-gather-7p59p"]
Sep 16 19:50:33.939793 ip-10-0-130-204 kubenswrapper[2565]: W0916 19:50:33.939765    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b12c0cf_66b3_48e1_b908_c7d1c6d47dd9.slice/crio-4a868c0de2cec8207f52ca10d4f5e653e87581869e99ead50e6530b34cc31495 WatchSource:0}: Error finding container 4a868c0de2cec8207f52ca10d4f5e653e87581869e99ead50e6530b34cc31495: Status 404 returned error can't find the container with id 4a868c0de2cec8207f52ca10d4f5e653e87581869e99ead50e6530b34cc31495
Sep 16 19:50:33.941838 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:33.941810    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:50:34.768147 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:34.768103    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s865n/must-gather-7p59p" event={"ID":"7b12c0cf-66b3-48e1-b908-c7d1c6d47dd9","Type":"ContainerStarted","Data":"4a868c0de2cec8207f52ca10d4f5e653e87581869e99ead50e6530b34cc31495"}
Sep 16 19:50:35.776253 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:35.776215    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s865n/must-gather-7p59p" event={"ID":"7b12c0cf-66b3-48e1-b908-c7d1c6d47dd9","Type":"ContainerStarted","Data":"15d3a60a9c86372f0441b573b59318a975eefce162d2d70fbd13979b7c29cd60"}
Sep 16 19:50:35.776681 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:35.776262    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s865n/must-gather-7p59p" event={"ID":"7b12c0cf-66b3-48e1-b908-c7d1c6d47dd9","Type":"ContainerStarted","Data":"7194fdbfd63a02b12e5eea2927a2af3267620e7d549165397bc11a010becc1a0"}
Sep 16 19:50:35.806031 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:35.805969    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s865n/must-gather-7p59p" podStartSLOduration=1.925234225 podStartE2EDuration="2.805950089s" totalImagesPullingTime="880.715864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:50:35.804261516 +0000 UTC m=+4274.163779511" watchObservedRunningTime="2026-09-16 19:50:35.805950089 +0000 UTC m=+4274.165468084"
Sep 16 19:50:36.453206 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:36.453161    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ddtnd_cea41c21-51ad-46c1-8f7c-fc4dab1008e7/global-pull-secret-syncer/0.log"
Sep 16 19:50:36.649076 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:36.649034    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t9j4z_dd830733-5618-401e-945e-21a97bb4b76f/konnectivity-agent/0.log"
Sep 16 19:50:36.679165 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:36.679107    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-204.ec2.internal_1152677edddc8ea58737be1eaabf066e/haproxy/0.log"
Sep 16 19:50:42.286970 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:42.286459    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-rkwxr_cd86797c-c8c8-4372-a44e-ca40a3c6a884/manager/0.log"
Sep 16 19:50:42.495127 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:42.494982    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-hf6t2_14eeb42f-1bd4-4b63-a462-487e6d1c7187/manager/0.log"
Sep 16 19:50:42.549126 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:42.548984    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-tgvjm_f356ad9e-45a9-4a92-9352-68ede7f9c77b/manager/0.log"
Sep 16 19:50:43.861581 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:43.861528    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_77b07526-8e5e-4d6d-915f-06ff138a9e8a/alertmanager/0.log"
Sep 16 19:50:43.897674 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:43.897634    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_77b07526-8e5e-4d6d-915f-06ff138a9e8a/config-reloader/0.log"
Sep 16 19:50:43.938278 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:43.938242    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_77b07526-8e5e-4d6d-915f-06ff138a9e8a/kube-rbac-proxy-web/0.log"
Sep 16 19:50:43.973052 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:43.973023    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_77b07526-8e5e-4d6d-915f-06ff138a9e8a/kube-rbac-proxy/0.log"
Sep 16 19:50:44.002707 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:44.002668    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_77b07526-8e5e-4d6d-915f-06ff138a9e8a/kube-rbac-proxy-metric/0.log"
Sep 16 19:50:44.039153 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:44.039125    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_77b07526-8e5e-4d6d-915f-06ff138a9e8a/prom-label-proxy/0.log"
Sep 16 19:50:44.062755 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:44.062715    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_77b07526-8e5e-4d6d-915f-06ff138a9e8a/init-config-reloader/0.log"
Sep 16 19:50:44.341923 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:44.341885    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-958gr_05cb61a7-1fb8-494f-a7f7-ad4c4808789f/node-exporter/0.log"
Sep 16 19:50:44.366290 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:44.366253    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-958gr_05cb61a7-1fb8-494f-a7f7-ad4c4808789f/kube-rbac-proxy/0.log"
Sep 16 19:50:44.394418 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:44.394381    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-958gr_05cb61a7-1fb8-494f-a7f7-ad4c4808789f/init-textfile/0.log"
Sep 16 19:50:44.608753 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:44.608717    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-jgd65_32481078-8fdf-47e9-b0f6-910736b236cf/kube-rbac-proxy-main/0.log"
Sep 16 19:50:44.645719 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:44.645682    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-jgd65_32481078-8fdf-47e9-b0f6-910736b236cf/kube-rbac-proxy-self/0.log"
Sep 16 19:50:44.668840 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:44.668807    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-jgd65_32481078-8fdf-47e9-b0f6-910736b236cf/openshift-state-metrics/0.log"
Sep 16 19:50:44.941974 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:44.941881    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-56dzj_d84d2a1b-19d2-41cf-8ff2-510c1e37705c/prometheus-operator-admission-webhook/0.log"
Sep 16 19:50:44.988450 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:44.988413    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69dd5589d6-tk55g_76b8bb64-3493-452f-9a4d-f1cc378be276/telemeter-client/0.log"
Sep 16 19:50:45.013070 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:45.013039    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69dd5589d6-tk55g_76b8bb64-3493-452f-9a4d-f1cc378be276/reload/0.log"
Sep 16 19:50:45.035022 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:45.034977    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69dd5589d6-tk55g_76b8bb64-3493-452f-9a4d-f1cc378be276/kube-rbac-proxy/0.log"
Sep 16 19:50:46.271494 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:46.271451    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-ts6zv_5acc4528-30cd-4e26-a485-02ba031ce861/networking-console-plugin/0.log"
Sep 16 19:50:46.974067 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:46.974011    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"]
Sep 16 19:50:46.979877 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:46.979836    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"
Sep 16 19:50:46.988746 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:46.988689    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"]
Sep 16 19:50:47.051704 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.051660    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d-podres\") pod \"perf-node-gather-daemonset-r2z7j\" (UID: \"ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"
Sep 16 19:50:47.051913 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.051810    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d-sys\") pod \"perf-node-gather-daemonset-r2z7j\" (UID: \"ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"
Sep 16 19:50:47.051913 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.051845    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d-lib-modules\") pod \"perf-node-gather-daemonset-r2z7j\" (UID: \"ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"
Sep 16 19:50:47.051913 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.051889    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d-proc\") pod \"perf-node-gather-daemonset-r2z7j\" (UID: \"ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"
Sep 16 19:50:47.052084 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.051984    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29qzb\" (UniqueName: \"kubernetes.io/projected/ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d-kube-api-access-29qzb\") pod \"perf-node-gather-daemonset-r2z7j\" (UID: \"ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"
Sep 16 19:50:47.153509 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.153458    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29qzb\" (UniqueName: \"kubernetes.io/projected/ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d-kube-api-access-29qzb\") pod \"perf-node-gather-daemonset-r2z7j\" (UID: \"ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"
Sep 16 19:50:47.153717 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.153567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d-podres\") pod \"perf-node-gather-daemonset-r2z7j\" (UID: \"ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"
Sep 16 19:50:47.153717 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.153638    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d-sys\") pod \"perf-node-gather-daemonset-r2z7j\" (UID: \"ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"
Sep 16 19:50:47.153717 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.153668    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d-lib-modules\") pod \"perf-node-gather-daemonset-r2z7j\" (UID: \"ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"
Sep 16 19:50:47.153907 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.153723    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d-proc\") pod \"perf-node-gather-daemonset-r2z7j\" (UID: \"ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"
Sep 16 19:50:47.153907 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.153759    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d-podres\") pod \"perf-node-gather-daemonset-r2z7j\" (UID: \"ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"
Sep 16 19:50:47.153907 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.153773    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d-sys\") pod \"perf-node-gather-daemonset-r2z7j\" (UID: \"ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"
Sep 16 19:50:47.153907 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.153840    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d-lib-modules\") pod \"perf-node-gather-daemonset-r2z7j\" (UID: \"ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"
Sep 16 19:50:47.153907 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.153845    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d-proc\") pod \"perf-node-gather-daemonset-r2z7j\" (UID: \"ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"
Sep 16 19:50:47.163009 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.162972    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29qzb\" (UniqueName: \"kubernetes.io/projected/ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d-kube-api-access-29qzb\") pod \"perf-node-gather-daemonset-r2z7j\" (UID: \"ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d\") " pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"
Sep 16 19:50:47.260975 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.260889    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-689959bb58-c2k5g_1b609927-192c-465e-b5a2-07b92efee24c/console/0.log"
Sep 16 19:50:47.303898 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.303857    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"
Sep 16 19:50:47.306947 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.306918    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-sbbgn_a4d4d25d-c795-416f-8398-d136b174cb36/download-server/0.log"
Sep 16 19:50:47.611995 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.611957    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"]
Sep 16 19:50:47.880214 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.880114    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j" event={"ID":"ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d","Type":"ContainerStarted","Data":"bada59f36be98e257037cbd8082fc49231863632f2846cdf4d3697e4ae58c801"}
Sep 16 19:50:47.880214 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.880179    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j" event={"ID":"ce2bbdc6-8e9e-4fdc-a027-54d9ebf7c76d","Type":"ContainerStarted","Data":"4db6dcc21d320db76a0a943333af3f5d1c556781a30e75b8d1ed88a29250a5e0"}
Sep 16 19:50:47.880476 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.880238    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"
Sep 16 19:50:47.908940 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:47.908867    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j" podStartSLOduration=1.908845217 podStartE2EDuration="1.908845217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:50:47.906546987 +0000 UTC m=+4286.266064997" watchObservedRunningTime="2026-09-16 19:50:47.908845217 +0000 UTC m=+4286.268363212"
Sep 16 19:50:49.079126 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:49.079097    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x2brk_359cd30d-7981-4184-8a6f-f63a2f585517/dns/0.log"
Sep 16 19:50:49.101785 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:49.101755    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x2brk_359cd30d-7981-4184-8a6f-f63a2f585517/kube-rbac-proxy/0.log"
Sep 16 19:50:49.188215 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:49.188181    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tch8d_fe676eb4-48c2-485b-9752-2198abad740a/dns-node-resolver/0.log"
Sep 16 19:50:49.729800 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:49.729751    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-68f546c58-hqpr7_4354a2be-2116-4f59-b84c-93060dd44d08/registry/0.log"
Sep 16 19:50:49.752963 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:49.752928    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8pj92_d1cdba36-53e5-400a-9fdd-7c62a3fe0086/node-ca/0.log"
Sep 16 19:50:50.894073 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:50.894040    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qg4vd_e2adc2cd-afaf-4150-8963-7d2b524b47e4/discovery/0.log"
Sep 16 19:50:51.622633 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:51.622588    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vb8t5_1a00e71a-7efb-45c5-ac52-6e175ea6837a/serve-healthcheck-canary/0.log"
Sep 16 19:50:52.343176 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:52.343126    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ww4bs_d1a62b24-aeeb-43c1-8279-3d4e479eb481/kube-rbac-proxy/0.log"
Sep 16 19:50:52.366040 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:52.366011    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ww4bs_d1a62b24-aeeb-43c1-8279-3d4e479eb481/exporter/0.log"
Sep 16 19:50:52.391247 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:52.391215    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ww4bs_d1a62b24-aeeb-43c1-8279-3d4e479eb481/extractor/0.log"
Sep 16 19:50:53.899538 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:53.899505    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s865n/perf-node-gather-daemonset-r2z7j"
Sep 16 19:50:56.338085 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:56.338048    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-557b59fdc4-f6qxg_c1126c57-2b37-4788-bb98-c043dde60b85/manager/0.log"
Sep 16 19:50:56.886829 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:56.886742    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6f9695cb4c-zvctb_c7f28c0c-962c-4b18-ad31-9fb9605873f5/manager/0.log"
Sep 16 19:50:56.991462 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:56.991410    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-b56cx_e246356d-16a5-4654-b0f8-0c1f36c085c9/server/0.log"
Sep 16 19:50:57.043827 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:57.043786    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qg4vd_e2adc2cd-afaf-4150-8963-7d2b524b47e4/discovery/0.log"
Sep 16 19:50:58.144731 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:58.144625    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-k88vm_e3ed332e-cd97-41ae-b43f-59c4b114d6b5/manager/0.log"
Sep 16 19:50:58.219127 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:58.219092    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-d825x_247f31d1-7a74-415f-8fdb-ea5289f128ac/seaweedfs/0.log"
Sep 16 19:50:58.447140 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:50:58.447057    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qg4vd_e2adc2cd-afaf-4150-8963-7d2b524b47e4/discovery/0.log"
Sep 16 19:51:03.035822 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:51:03.035790    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-9tgwr_a266096e-4a22-411c-a43f-2d68a3dd93ed/migrator/0.log"
Sep 16 19:51:03.056993 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:51:03.056959    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-9tgwr_a266096e-4a22-411c-a43f-2d68a3dd93ed/graceful-termination/0.log"
Sep 16 19:51:04.457471 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:51:04.457431    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8dncf_2dce3890-5613-4464-91b5-388c8579cfee/kube-multus/0.log"
Sep 16 19:51:04.482703 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:51:04.482674    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hhl4g_f251ea42-1492-4bf4-942e-fdd522e5bbc5/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:51:04.510233 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:51:04.510203    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hhl4g_f251ea42-1492-4bf4-942e-fdd522e5bbc5/egress-router-binary-copy/0.log"
Sep 16 19:51:04.539604 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:51:04.539573    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hhl4g_f251ea42-1492-4bf4-942e-fdd522e5bbc5/cni-plugins/0.log"
Sep 16 19:51:04.564958 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:51:04.564926    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hhl4g_f251ea42-1492-4bf4-942e-fdd522e5bbc5/bond-cni-plugin/0.log"
Sep 16 19:51:04.586371 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:51:04.586322    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hhl4g_f251ea42-1492-4bf4-942e-fdd522e5bbc5/routeoverride-cni/0.log"
Sep 16 19:51:04.608153 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:51:04.608124    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hhl4g_f251ea42-1492-4bf4-942e-fdd522e5bbc5/whereabouts-cni-bincopy/0.log"
Sep 16 19:51:04.630734 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:51:04.630694    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hhl4g_f251ea42-1492-4bf4-942e-fdd522e5bbc5/whereabouts-cni/0.log"
Sep 16 19:51:05.064080 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:51:05.064049    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7r6xr_2344014b-38cc-448f-9c9d-6b49d8e80b85/network-metrics-daemon/0.log"
Sep 16 19:51:05.081207 ip-10-0-130-204 kubenswrapper[2565]: I0916 19:51:05.081175    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7r6xr_2344014b-38cc-448f-9c9d-6b49d8e80b85/kube-rbac-proxy/0.log"