Jul 28 01:25:55.127976 ip-10-0-139-81 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 01:25:56.717789 ip-10-0-139-81 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:25:56.717789 ip-10-0-139-81 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 01:25:56.717789 ip-10-0-139-81 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:25:56.717789 ip-10-0-139-81 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 01:25:56.718390 ip-10-0-139-81 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:25:56.719008 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.718923    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 01:25:56.723482 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723465    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:56.723482 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723482    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723486    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723489    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723493    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723496    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723506    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723509    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723512    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723514    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723517    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723520    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723522    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723525    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723527    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723530    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723533    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723536    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723539    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723541    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723544    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:56.723561 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723546    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:56.724033 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723549    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:56.724033 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723551    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:56.724033 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723553    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:56.724033 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723556    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:56.724033 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723559    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:56.724033 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723562    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:56.724033 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723564    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:56.724033 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723567    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:56.724033 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723576    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:56.724033 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723579    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:56.724033 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723582    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:56.724033 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723586    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:56.724033 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723590    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:56.724033 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723593    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:56.724033 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723596    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:56.724033 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723599    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:56.724033 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723601    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:56.724033 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723603    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:56.724033 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723606    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:56.724509 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723608    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:56.724509 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723611    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:56.724509 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723613    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:56.724509 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723617    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:56.724509 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723620    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:56.724509 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723623    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:56.724509 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723625    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:56.724509 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723628    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:56.724509 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723631    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:56.724509 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723634    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:56.724509 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723636    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:56.724509 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723639    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:56.724509 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723641    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:56.724509 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723644    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:56.724509 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723648    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:56.724509 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723651    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:56.724509 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723654    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:56.724509 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723657    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:56.724509 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723660    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723662    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723665    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723669    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723674    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723676    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723679    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723681    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723684    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723687    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723689    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723692    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723694    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723697    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723700    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723702    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723705    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723707    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723710    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723713    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:56.724971 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723716    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723719    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723721    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723724    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723727    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723729    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723732    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723735    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723738    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723741    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723744    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723747    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723749    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723752    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723755    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723757    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723760    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723763    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723765    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.723768    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:56.725495 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724499    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:56.725982 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724508    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:56.725982 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724513    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:56.725982 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724516    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:56.725982 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724520    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:56.725982 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724524    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:56.725982 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724527    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:56.725982 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724530    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:56.725982 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724533    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:56.725982 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724536    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:56.725982 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724539    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:56.725982 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724541    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:56.725982 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724544    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:56.725982 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724547    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:56.725982 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724549    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:56.725982 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724552    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:56.725982 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724554    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:56.725982 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724557    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:56.725982 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724560    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:56.725982 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724563    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:56.726474 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724566    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:56.726474 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724569    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:56.726474 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724571    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:56.726474 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724574    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:56.726474 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724577    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:56.726474 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724579    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:56.726474 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724582    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:56.726474 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724586    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:56.726474 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724589    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:56.726474 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724592    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:56.726474 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724595    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:56.726474 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724597    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:56.726474 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724600    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:56.726474 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724603    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:56.726474 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724606    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:56.726474 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724608    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:56.726474 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724611    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:56.726474 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724613    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:56.726474 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724618    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724621    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724624    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724627    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724630    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724632    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724635    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724638    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724640    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724643    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724646    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724648    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724651    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724654    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724657    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724659    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724662    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724664    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724667    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724669    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:56.726930 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724672    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724674    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724676    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724679    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724682    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724685    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724688    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724690    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724693    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724695    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724698    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724700    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724703    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724705    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724708    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724710    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724713    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724715    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724718    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724720    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:56.727443 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724723    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724725    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724728    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724730    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724734    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724736    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724740    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724743    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724745    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724748    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724751    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724753    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724756    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724759    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724761    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724764    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724766    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724769    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724772    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724774    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:56.727948 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.724777    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725146    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725150    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725153    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725156    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725158    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725161    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725164    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725166    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725169    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725172    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725174    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725177    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725179    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725182    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725184    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725187    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725190    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725193    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725195    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:56.728467 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725198    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:56.728960 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725200    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:56.728960 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725203    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:56.728960 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725205    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:56.728960 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725208    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:56.728960 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725210    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:56.728960 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725213    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:56.728960 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725215    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:56.728960 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725218    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:56.728960 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725220    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:56.728960 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725223    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:56.728960 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725226    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:56.728960 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725229    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:56.728960 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725232    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:56.728960 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725234    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:56.728960 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725237    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:56.728960 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725240    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:56.728960 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725242    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:56.728960 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725244    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:56.728960 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725247    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725250    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725252    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725254    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725257    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725259    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725262    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725264    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725267    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725270    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725273    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725275    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725278    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725281    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725283    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725286    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725288    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725291    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725293    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725296    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:56.729421 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725298    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725301    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725303    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725305    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725308    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725311    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725314    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725316    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725319    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725321    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725325    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725342    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725345    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725347    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725350    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725352    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725355    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725357    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725360    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725363    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:56.729950 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725368    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:56.730520 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725371    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:56.730520 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725374    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:56.730520 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725377    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:56.730520 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725379    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:56.730520 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725382    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:56.730520 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725385    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:56.730520 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725387    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:56.730520 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725390    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:56.730520 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725392    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:56.730520 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725395    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:56.730520 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725397    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:56.730520 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725400    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:56.730520 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725405    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:56.730520 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725409    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:56.730520 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725412    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:56.730520 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725415    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:56.730520 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725419    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:56.730520 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725422    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:56.730520 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725424    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.725427    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725497    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725504    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725511    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725516    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725520    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725523    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725528    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725532    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725535    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725538    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725542    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725547    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725550    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725553    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725557    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725560    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725563    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725566    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725570    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725573    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725577    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725580    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 01:25:56.730980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725584    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725587    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725591    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725594    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725597    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725600    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725603    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725607    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725610    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725613    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725617    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725621    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725624    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725626    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725629    2578 flags.go:64] FLAG: --enable-server="true"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725632    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725636    2578 flags.go:64] FLAG: --event-burst="100"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725640    2578 flags.go:64] FLAG: --event-qps="50"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725642    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725645    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725650    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725655    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725658    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725661    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725664    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 28 01:25:56.731611 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725668    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725670    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725673    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725676    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725679    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725682    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725685    2578 flags.go:64] FLAG: --feature-gates=""
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725688    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725691    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725694    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725697    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725700    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725703    2578 flags.go:64] FLAG: --help="false"
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725706    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-139-81.ec2.internal"
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725710    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725714    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725716    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725720    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725723    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725726    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725729    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725731    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725734    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 01:25:56.732202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725737    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725740    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725743    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725746    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725750    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725755    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725757    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725760    2578 flags.go:64] FLAG: --lock-file=""
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725763    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725766    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725769    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725775    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725777    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725780    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725783    2578 flags.go:64] FLAG: --logging-format="text"
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725786    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725789    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725792    2578 flags.go:64] FLAG: --manifest-url=""
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725795    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725800    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725803    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725807    2578 flags.go:64] FLAG: --max-pods="110"
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725810    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725813    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725816    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 01:25:56.732831 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725819    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725822    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725825    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725828    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725834    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725837    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725840    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725843    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725846    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725851    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725854    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725858    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725864    2578 flags.go:64] FLAG: --port="10250"
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725867    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725870    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0efbf2c5fe89efc17"
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725873    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725876    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725879    2578 flags.go:64] FLAG: --register-node="true"
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725882    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725886    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725889    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725891    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725894    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725898    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725901    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 01:25:56.733470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725904    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725906    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725909    2578 flags.go:64] FLAG: --runonce="false"
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725912    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725916    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725919    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725922    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725925    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725928    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725931    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725934    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725937    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725940    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725943    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725946    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725949    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725951    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725954    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725959    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725964    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725967    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725972    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725975    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725977    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725980    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 01:25:56.734108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725983    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725987    2578 flags.go:64] FLAG: --v="2"
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725991    2578 flags.go:64] FLAG: --version="false"
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725995    2578 flags.go:64] FLAG: --vmodule=""
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.725999    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.726002    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726114    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726117    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726120    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726123    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726125    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726128    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726131    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726134    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726136    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726139    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726142    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726145    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726147    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726150    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726152    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:56.734737 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726155    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:56.735251 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726157    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:56.735251 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726160    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:56.735251 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726163    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:56.735251 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726167    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:56.735251 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726174    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:56.735251 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726177    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:56.735251 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726180    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:56.735251 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726183    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:56.735251 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726185    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:56.735251 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726188    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:56.735251 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726190    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:56.735251 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726193    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:56.735251 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726196    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:56.735251 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726199    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:56.735251 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726201    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:56.735251 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726205    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:56.735251 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726208    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:56.735251 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726211    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:56.735251 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726214    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726217    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726219    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726222    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726224    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726227    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726229    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726232    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726234    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726237    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726240    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726242    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726245    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726248    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726250    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726252    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726255    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726257    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726263    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726265    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:56.735818 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726268    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:56.736301 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726271    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:56.736301 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726273    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:56.736301 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726276    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:56.736301 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726278    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:56.736301 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726281    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:56.736301 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726283    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:56.736301 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726286    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:56.736301 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726289    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:56.736301 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726291    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:56.736301 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726294    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:56.736301 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726296    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:56.736301 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726299    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:56.736301 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726302    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:56.736301 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726304    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:56.736301 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726306    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:56.736301 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726309    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:56.736301 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726311    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:56.736301 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726314    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:56.736301 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726317    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726320    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726324    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726342    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726347    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726351    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726355    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726359    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726362    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726364    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726367    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726375    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726378    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726380    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726383    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726386    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726388    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726391    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726393    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726396    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:56.736775 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726398    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:56.737257 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726401    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:56.737257 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726403    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:56.737257 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726406    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:56.737257 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726408    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:56.737257 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.726411    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:56.737257 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.727040    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:25:56.737257 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.734809    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 01:25:56.737257 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.734825    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 01:25:56.737257 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.735715    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 01:25:56.737257 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.735724    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 01:25:56.737257 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735783    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:56.737257 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735789    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:56.737257 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735792    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:56.737257 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735797    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735800    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735803    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735806    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735809    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735812    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735814    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735817    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735820    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735822    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735825    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735828    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735831    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735834    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735836    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735839    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735841    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735844    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735847    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735849    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:56.737625 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735852    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:56.738109 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735854    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:56.738109 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735857    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:56.738109 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735860    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:56.738109 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735863    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:56.738109 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735866    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:56.738109 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735871    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:56.738109 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735875    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:56.738109 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735878    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:56.738109 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735881    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:56.738109 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735884    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:56.738109 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735887    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:56.738109 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735890    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:56.738109 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735893    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:56.738109 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735896    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:56.738109 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735900    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:56.738109 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735903    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:56.738109 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735906    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:56.738109 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735909    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:56.738109 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735911    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735914    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735917    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735919    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735922    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735925    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735927    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735930    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735933    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735935    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735938    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735941    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735944    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735946    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735950    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735954    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735957    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735960    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735963    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735966    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:56.738591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735969    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735972    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735975    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735977    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735980    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735982    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735985    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735987    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735990    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735992    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735995    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.735997    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736000    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736002    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736005    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736007    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736010    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736012    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736015    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736017    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:56.739066 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736020    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:56.739681 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736022    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:56.739681 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736025    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:56.739681 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736027    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:56.739681 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736030    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:56.739681 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736032    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:56.739681 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736035    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:56.739681 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736038    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:56.739681 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736040    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:56.739681 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736043    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:56.739681 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736045    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:56.739681 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736048    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:56.739681 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736050    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:56.739681 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736054    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:56.739681 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736056    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:56.739681 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736059    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:56.739681 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736061    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:56.740070 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.736066    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:25:56.740070 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736182    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:56.740070 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736185    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:56.740070 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736188    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:56.740070 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736191    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:56.740070 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736194    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:56.740070 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736196    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:56.740070 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736199    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:56.740070 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736202    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:56.740070 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736204    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:56.740070 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736207    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:56.740070 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736209    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:56.740070 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736212    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:56.740070 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736214    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:56.740070 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736217    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736220    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736222    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736225    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736227    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736229    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736232    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736235    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736237    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736240    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736242    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736245    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736247    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736250    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736252    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736255    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736258    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736261    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736263    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736266    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:56.740456 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736268    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:56.740941 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736270    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:56.740941 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736274    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:56.740941 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736278    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:56.740941 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736281    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:56.740941 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736283    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:56.740941 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736286    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:56.740941 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736288    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:56.740941 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736291    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:56.740941 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736293    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:56.740941 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736296    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:56.740941 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736298    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:56.740941 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736301    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:56.740941 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736303    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:56.740941 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736306    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:56.740941 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736309    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:56.740941 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736311    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:56.740941 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736314    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:56.740941 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736317    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:56.740941 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736319    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:56.741423 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736322    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:56.741423 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736324    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:56.741423 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736344    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:56.741423 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736349    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:56.741423 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736353    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:56.741423 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736356    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:56.741423 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736359    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:56.741423 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736361    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:56.741423 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736365    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:56.741423 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736367    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:56.741423 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736370    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:56.741423 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736372    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:56.741423 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736375    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:56.741423 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736378    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:56.741423 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736380    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:56.741423 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736383    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:56.741423 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736385    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:56.741423 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736387    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:56.741423 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736390    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736392    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736395    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736397    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736400    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736402    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736405    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736407    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736410    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736412    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736415    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736417    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736420    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736422    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736425    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736428    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736430    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736433    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736435    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736438    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:56.741880 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736441    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:56.742373 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736443    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:56.742373 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736446    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:56.742373 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736449    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:56.742373 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736452    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:56.742373 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736454    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:56.742373 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736457    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:56.742373 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:56.736459    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:56.742373 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.736464    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:25:56.742373 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.736570    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 01:25:56.742373 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.738702    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 01:25:56.742373 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.740235    2578 server.go:1019] "Starting client certificate rotation"
Jul 28 01:25:56.742373 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.740346    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 01:25:56.742373 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.741365    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 01:25:56.742779 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.742766    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 01:25:56.750489 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.750469    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 01:25:56.759143 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.759126    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 28 01:25:56.761422 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.761406    2578 log.go:25] "Validated CRI v1 image API"
Jul 28 01:25:56.762199 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.762184    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 01:25:56.765805 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.765779    2578 fs.go:135] Filesystem UUIDs: map[09187b8e-1413-4420-b03a-0947e9add7c8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f18521f4-a819-4007-a342-50c1b092f5e1:/dev/nvme0n1p3]
Jul 28 01:25:56.765882 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.765804    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 01:25:56.771285 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.771168    2578 manager.go:216] Machine: {Timestamp:2026-07-28 01:25:56.770689986 +0000 UTC m=+0.348147599 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100474 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d99136f14c3180f136ec8b5225c8e SystemUUID:ec2d9913-6f14-c318-0f13-6ec8b5225c8e BootID:6baf5690-6b1e-4c14-a6f3-e5692150fd00 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:93:e0:63:af:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:93:e0:63:af:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:8c:3b:30:ce:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 01:25:56.771285 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.771275    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 01:25:56.771429 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.771368    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 01:25:56.771673 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.771657    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 01:25:56.774861 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.774834    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 01:25:56.774992 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.774864    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-81.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 01:25:56.775694 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.775683    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 01:25:56.775731 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.775696    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 01:25:56.775731 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.775704    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 01:25:56.775731 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.775718    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 01:25:56.776592 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.776582    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 01:25:56.778473 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.778462    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 01:25:56.778592 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.778582    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 01:25:56.778629 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.778610    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 01:25:56.778629 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.778620    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 01:25:56.778703 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.778639    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 01:25:56.778703 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.778649    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 28 01:25:56.778703 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.778663    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 01:25:56.779045 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.779033    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 01:25:56.779078 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.779055    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 01:25:56.782851 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.782823    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 01:25:56.784959 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.784946    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 01:25:56.785005 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.784977    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 01:25:56.786777 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.786766    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 01:25:56.786824 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.786781    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 01:25:56.786824 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.786788    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 01:25:56.786824 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.786793    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 01:25:56.786824 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.786802    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 01:25:56.786824 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.786807    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 01:25:56.786824 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.786820    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 01:25:56.786824 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.786827    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 01:25:56.787009 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.786835    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 01:25:56.787009 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.786840    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 01:25:56.787009 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.786853    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 01:25:56.787009 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.786860    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 01:25:56.788212 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:56.788188    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-81.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 01:25:56.788836 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.788819    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 01:25:56.788964 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:56.788932    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 01:25:56.789756 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.789738    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 01:25:56.792922 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.792903    2578 server.go:1268] "Started kubelet"
Jul 28 01:25:56.793008 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.792983    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 01:25:56.793072 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.792994    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 01:25:56.793072 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.793068    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 01:25:56.793900 ip-10-0-139-81 systemd[1]: Started Kubernetes Kubelet.
Jul 28 01:25:56.795114 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.795095    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 01:25:56.795931 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.795914    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hlgwv"
Jul 28 01:25:56.797837 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.797784    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 01:25:56.797837 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:56.797813    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 01:25:56.801149 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.801130    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-81.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 01:25:56.802172 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:56.801204    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-81.ec2.internal.18c64fb8a6f88f4b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-81.ec2.internal,UID:ip-10-0-139-81.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-81.ec2.internal,},FirstTimestamp:2026-07-28 01:25:56.792864587 +0000 UTC m=+0.370322195,LastTimestamp:2026-07-28 01:25:56.792864587 +0000 UTC m=+0.370322195,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-81.ec2.internal,}"
Jul 28 01:25:56.804104 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.804089    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 01:25:56.804104 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.804097    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 01:25:56.804219 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:56.804166    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Jul 28 01:25:56.804751 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.804733    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 01:25:56.804751 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.804737    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 01:25:56.804865 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.804764    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 01:25:56.804913 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.804866    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 01:25:56.804913 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.804868    2578 factory.go:55] Registering systemd factory
Jul 28 01:25:56.804913 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.804886    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 28 01:25:56.804913 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.804873    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 01:25:56.805098 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.804986    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hlgwv"
Jul 28 01:25:56.805098 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.805096    2578 factory.go:153] Registering CRI-O factory
Jul 28 01:25:56.805188 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.805108    2578 factory.go:223] Registration of the crio container factory successfully
Jul 28 01:25:56.805188 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.805154    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 01:25:56.805188 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.805175    2578 factory.go:103] Registering Raw factory
Jul 28 01:25:56.805188 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.805189    2578 manager.go:1196] Started watching for new ooms in manager
Jul 28 01:25:56.805858 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.805844    2578 manager.go:349] Starting recovery of all containers
Jul 28 01:25:56.807279 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:56.807066    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 01:25:56.807279 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:56.807165    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-81.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 01:25:56.816792 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.816618    2578 manager.go:354] Recovery completed
Jul 28 01:25:56.818093 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:56.818074    2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 28 01:25:56.822510 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.822473    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:56.824405 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.824388    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:56.824470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.824418    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:56.824470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.824431    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:56.824966 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.824951    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 01:25:56.824966 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.824963    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 01:25:56.825080 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.824981    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 01:25:56.825080 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.825008    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:56.825686 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.825673    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:56.825741 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.825700    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:56.825741 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.825715    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:56.826984 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.826973    2578 policy_none.go:49] "None policy: Start"
Jul 28 01:25:56.827043 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.826988    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 01:25:56.827043 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.826999    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 01:25:56.828070 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.828058    2578 policy_none.go:47] "Start"
Jul 28 01:25:56.866879 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.866866    2578 manager.go:337] "Starting Device Plugin manager"
Jul 28 01:25:56.866963 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:56.866894    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 01:25:56.866963 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.866904    2578 server.go:87] "Starting device plugin registration server"
Jul 28 01:25:56.867148 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.867136    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 01:25:56.867214 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.867151    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 01:25:56.867264 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.867240    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 01:25:56.867388 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.867325    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 01:25:56.867388 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.867350    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 01:25:56.867820 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:56.867802    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 01:25:56.867890 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:56.867851    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-81.ec2.internal\" not found"
Jul 28 01:25:56.967596 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.967534    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:56.968851 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.968723    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:56.968851 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.968752    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:56.968851 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.968762    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:56.968851 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.968788    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-81.ec2.internal"
Jul 28 01:25:56.976985 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.976965    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-81.ec2.internal"
Jul 28 01:25:56.977093 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:56.976994    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-81.ec2.internal\": node \"ip-10-0-139-81.ec2.internal\" not found"
Jul 28 01:25:56.977679 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.977661    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 01:25:56.979139 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.979109    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 01:25:56.979221 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.979159    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 01:25:56.979221 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.979184    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 01:25:56.979221 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:56.979214    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 01:25:56.981488 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:56.981469    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:57.025845 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:57.025813    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Jul 28 01:25:57.079368 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.079318    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal"]
Jul 28 01:25:57.079446 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.079409    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:57.080469 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.080451    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:57.080530 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.080494    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:57.080530 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.080509    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:57.081975 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.081962    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:57.082122 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.082106    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal"
Jul 28 01:25:57.082172 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.082137    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:57.082746 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.082730    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:57.082826 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.082759    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:57.082826 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.082772    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:57.083303 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.083291    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:57.083369 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.083320    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:57.083369 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.083342    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:57.084124 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.084110    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal"
Jul 28 01:25:57.084168 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.084135    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:57.084861 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.084841    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:57.084968 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.084868    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:57.084968 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.084881    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:57.106333 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.106313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/216527ecc9658544f9bc7b3b7800c13c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-81.ec2.internal\" (UID: \"216527ecc9658544f9bc7b3b7800c13c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal"
Jul 28 01:25:57.106441 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.106355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4fa6e8ab638aa3b95e5e62c2f5cb6258-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal\" (UID: \"4fa6e8ab638aa3b95e5e62c2f5cb6258\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal"
Jul 28 01:25:57.106441 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.106377    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fa6e8ab638aa3b95e5e62c2f5cb6258-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal\" (UID: \"4fa6e8ab638aa3b95e5e62c2f5cb6258\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal"
Jul 28 01:25:57.109938 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:57.109921    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-81.ec2.internal\" not found" node="ip-10-0-139-81.ec2.internal"
Jul 28 01:25:57.114189 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:57.114172    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-81.ec2.internal\" not found" node="ip-10-0-139-81.ec2.internal"
Jul 28 01:25:57.126411 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:57.126393    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Jul 28 01:25:57.207132 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.207106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/216527ecc9658544f9bc7b3b7800c13c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-81.ec2.internal\" (UID: \"216527ecc9658544f9bc7b3b7800c13c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal"
Jul 28 01:25:57.207132 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.207135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4fa6e8ab638aa3b95e5e62c2f5cb6258-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal\" (UID: \"4fa6e8ab638aa3b95e5e62c2f5cb6258\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal"
Jul 28 01:25:57.207344 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.207155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fa6e8ab638aa3b95e5e62c2f5cb6258-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal\" (UID: \"4fa6e8ab638aa3b95e5e62c2f5cb6258\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal"
Jul 28 01:25:57.207344 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.207208    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fa6e8ab638aa3b95e5e62c2f5cb6258-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal\" (UID: \"4fa6e8ab638aa3b95e5e62c2f5cb6258\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal"
Jul 28 01:25:57.207344 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.207223    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/216527ecc9658544f9bc7b3b7800c13c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-81.ec2.internal\" (UID: \"216527ecc9658544f9bc7b3b7800c13c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal"
Jul 28 01:25:57.207344 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.207235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4fa6e8ab638aa3b95e5e62c2f5cb6258-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal\" (UID: \"4fa6e8ab638aa3b95e5e62c2f5cb6258\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal"
Jul 28 01:25:57.227260 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:57.227205    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Jul 28 01:25:57.328127 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:57.328079    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Jul 28 01:25:57.413608 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.413577    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal"
Jul 28 01:25:57.417923 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.417856    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal"
Jul 28 01:25:57.428634 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:57.428615    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Jul 28 01:25:57.529151 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:57.529117    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Jul 28 01:25:57.629638 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:57.629611    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Jul 28 01:25:57.730264 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:57.730231    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Jul 28 01:25:57.739424 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.739399    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 01:25:57.739577 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.739557    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 01:25:57.809690 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.809609    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 01:20:56 +0000 UTC" deadline="2028-05-04 07:42:16.133560582 +0000 UTC"
Jul 28 01:25:57.809690 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:57.809639    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15510h16m18.323925069s"
Jul 28 01:25:57.830810 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:57.830780    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Jul 28 01:25:57.931421 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:57.931378    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Jul 28 01:25:58.000705 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:58.000659    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216527ecc9658544f9bc7b3b7800c13c.slice/crio-6ea7135b723ff52ff21e3f5f2f630a5f88be12dce600c75e8cfc540707e43100 WatchSource:0}: Error finding container 6ea7135b723ff52ff21e3f5f2f630a5f88be12dce600c75e8cfc540707e43100: Status 404 returned error can't find the container with id 6ea7135b723ff52ff21e3f5f2f630a5f88be12dce600c75e8cfc540707e43100
Jul 28 01:25:58.001088 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:58.001068    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa6e8ab638aa3b95e5e62c2f5cb6258.slice/crio-580847dd2337b1aa40c8c8744fd59ab84d1fd9e1a071f0d998863521b79d9511 WatchSource:0}: Error finding container 580847dd2337b1aa40c8c8744fd59ab84d1fd9e1a071f0d998863521b79d9511: Status 404 returned error can't find the container with id 580847dd2337b1aa40c8c8744fd59ab84d1fd9e1a071f0d998863521b79d9511
Jul 28 01:25:58.006766 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.006749    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:25:58.032321 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:58.032289    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-81.ec2.internal\" not found"
Jul 28 01:25:58.032591 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.032575    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:58.111246 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.111174    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:58.204541 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.204502    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal"
Jul 28 01:25:58.214768 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.214741    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 01:25:58.215810 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.215795    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal"
Jul 28 01:25:58.227730 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.227706    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 01:25:58.406552 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.406466    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:58.780255 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.780219    2578 apiserver.go:52] "Watching apiserver"
Jul 28 01:25:58.791208 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.791080    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 01:25:58.792226 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.792191    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-wpcdj","kube-system/konnectivity-agent-c9ntv","kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal","openshift-dns/node-resolver-wlzmw","openshift-image-registry/node-ca-2d5z9","openshift-multus/multus-qqq2z","openshift-multus/network-metrics-daemon-dx5sm","openshift-network-diagnostics/network-check-target-zzc9f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t","openshift-cluster-node-tuning-operator/tuned-bh7l6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal","openshift-multus/multus-additional-cni-plugins-gwwvx","openshift-network-operator/iptables-alerter-qnv59","openshift-ovn-kubernetes/ovnkube-node-l5mxg"]
Jul 28 01:25:58.793862 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.793833    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2d5z9"
Jul 28 01:25:58.795187 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.795167    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.796843 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.796822    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pbjmw\""
Jul 28 01:25:58.797353 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.797127    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 01:25:58.797353 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.797157    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:25:58.797353 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.797162    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 01:25:58.797353 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:58.797227    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dx5sm" podUID="ddcb15a3-037d-45a0-8787-12585c0d2465"
Jul 28 01:25:58.797599 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.797506    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 01:25:58.798497 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.798480    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wlzmw"
Jul 28 01:25:58.800001 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.799982    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.801512 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.801414    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:25:58.801512 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:58.801479    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpcdj" podUID="cd1a9e35-4116-41f7-b275-9da51eb2549a"
Jul 28 01:25:58.802953 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.802813    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:25:58.802953 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.802836    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 01:25:58.802953 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.802865    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 01:25:58.802953 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:58.802914    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzc9f" podUID="e5ba3944-6409-4af9-9bff-8eb91a65755a"
Jul 28 01:25:58.804492 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.804469    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c9ntv"
Jul 28 01:25:58.804720 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.804697    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 01:25:58.806185 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.806040    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.806870 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.806847    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 01:25:58.807241 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.807221    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:58.808766 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.808746    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qnv59"
Jul 28 01:25:58.809160 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.809135    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 01:25:58.809767 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.809315    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kpl6s\""
Jul 28 01:25:58.809767 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.809385    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 01:25:58.809767 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.809458    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 01:25:58.809767 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.809491    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 01:25:58.809767 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.809560    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 01:25:58.809767 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.809680    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 01:25:58.809767 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.809687    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 01:25:58.811418 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.810511    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 01:25:58.811418 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.810584    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pgqqx\""
Jul 28 01:25:58.811418 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.810625    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:25:58.811418 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.810700    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wqv29\""
Jul 28 01:25:58.811418 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.810734    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-58d66\""
Jul 28 01:25:58.811418 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.810509    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.811418 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.810802    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 01:25:58.811418 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.810865    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 01:25:58.811418 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.810941    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4xr5b\""
Jul 28 01:25:58.811418 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.810950    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 01:25:58.811418 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.811191    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:25:58.811418 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.811226    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-prmw9\""
Jul 28 01:25:58.815971 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.815949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-systemd\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.816067 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.815987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z429\" (UniqueName: \"kubernetes.io/projected/640c683a-7ed6-4686-a0de-f6c631e1ab3a-kube-api-access-5z429\") pod \"node-resolver-wlzmw\" (UID: \"640c683a-7ed6-4686-a0de-f6c631e1ab3a\") " pod="openshift-dns/node-resolver-wlzmw"
Jul 28 01:25:58.816067 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816014    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-host-var-lib-cni-bin\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.816067 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-host-var-lib-kubelet\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.816067 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816064    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-hostroot\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.816274 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wcxc\" (UniqueName: \"kubernetes.io/projected/b8193a25-2add-42b3-8815-022a163695fe-kube-api-access-4wcxc\") pod \"node-ca-2d5z9\" (UID: \"b8193a25-2add-42b3-8815-022a163695fe\") " pod="openshift-image-registry/node-ca-2d5z9"
Jul 28 01:25:58.816274 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816111    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/05bfa172-3907-43d0-a248-b4c74dd7620d-registration-dir\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.816274 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret\") pod \"global-pull-secret-syncer-wpcdj\" (UID: \"cd1a9e35-4116-41f7-b275-9da51eb2549a\") " pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:25:58.816274 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-run\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.816274 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-host-run-k8s-cni-cncf-io\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.816274 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816197    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-host-run-multus-certs\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.816274 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2ddg\" (UniqueName: \"kubernetes.io/projected/65e2f293-2ed4-4ff1-aba1-ca582bde2904-kube-api-access-h2ddg\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.816274 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816258    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-sysctl-conf\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.816679 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-sys\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.816679 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-lib-modules\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.816679 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-tuned\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.816679 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816375    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmmbx\" (UniqueName: \"kubernetes.io/projected/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-kube-api-access-tmmbx\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.816679 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fbts\" (UniqueName: \"kubernetes.io/projected/ddcb15a3-037d-45a0-8787-12585c0d2465-kube-api-access-4fbts\") pod \"network-metrics-daemon-dx5sm\" (UID: \"ddcb15a3-037d-45a0-8787-12585c0d2465\") " pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:25:58.816679 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/640c683a-7ed6-4686-a0de-f6c631e1ab3a-hosts-file\") pod \"node-resolver-wlzmw\" (UID: \"640c683a-7ed6-4686-a0de-f6c631e1ab3a\") " pod="openshift-dns/node-resolver-wlzmw"
Jul 28 01:25:58.816679 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/05bfa172-3907-43d0-a248-b4c74dd7620d-socket-dir\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.816679 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816491    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/05bfa172-3907-43d0-a248-b4c74dd7620d-etc-selinux\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.816679 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd1a9e35-4116-41f7-b275-9da51eb2549a-kubelet-config\") pod \"global-pull-secret-syncer-wpcdj\" (UID: \"cd1a9e35-4116-41f7-b275-9da51eb2549a\") " pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:25:58.816679 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd1a9e35-4116-41f7-b275-9da51eb2549a-dbus\") pod \"global-pull-secret-syncer-wpcdj\" (UID: \"cd1a9e35-4116-41f7-b275-9da51eb2549a\") " pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:25:58.816679 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-kubernetes\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.816679 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-var-lib-kubelet\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.816679 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8193a25-2add-42b3-8815-022a163695fe-host\") pod \"node-ca-2d5z9\" (UID: \"b8193a25-2add-42b3-8815-022a163695fe\") " pod="openshift-image-registry/node-ca-2d5z9"
Jul 28 01:25:58.817295 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-os-release\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.817295 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816718    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65e2f293-2ed4-4ff1-aba1-ca582bde2904-cni-binary-copy\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.817295 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-host-var-lib-cni-multus\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.817295 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816795    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/05bfa172-3907-43d0-a248-b4c74dd7620d-device-dir\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.817295 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-sysctl-d\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.817295 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-sysconfig\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.817295 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816878    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-multus-cni-dir\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.817295 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-multus-conf-dir\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.817295 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/65e2f293-2ed4-4ff1-aba1-ca582bde2904-multus-daemon-config\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.817295 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/05bfa172-3907-43d0-a248-b4c74dd7620d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.817295 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/05bfa172-3907-43d0-a248-b4c74dd7620d-sys-fs\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.817295 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.816990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf7mk\" (UniqueName: \"kubernetes.io/projected/05bfa172-3907-43d0-a248-b4c74dd7620d-kube-api-access-bf7mk\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.817295 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.817012    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-host\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.817295 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.817035    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-tmp\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.817295 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.817061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs\") pod \"network-metrics-daemon-dx5sm\" (UID: \"ddcb15a3-037d-45a0-8787-12585c0d2465\") " pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:25:58.817295 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.817097    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-cnibin\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.817295 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.817148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-host-run-netns\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.818232 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.817169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-etc-kubernetes\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.818232 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.817212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/069b6671-6207-471b-a09c-26a0293ba1ba-konnectivity-ca\") pod \"konnectivity-agent-c9ntv\" (UID: \"069b6671-6207-471b-a09c-26a0293ba1ba\") " pod="kube-system/konnectivity-agent-c9ntv"
Jul 28 01:25:58.818232 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.817234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/069b6671-6207-471b-a09c-26a0293ba1ba-agent-certs\") pod \"konnectivity-agent-c9ntv\" (UID: \"069b6671-6207-471b-a09c-26a0293ba1ba\") " pod="kube-system/konnectivity-agent-c9ntv"
Jul 28 01:25:58.818232 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.817257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/640c683a-7ed6-4686-a0de-f6c631e1ab3a-tmp-dir\") pod \"node-resolver-wlzmw\" (UID: \"640c683a-7ed6-4686-a0de-f6c631e1ab3a\") " pod="openshift-dns/node-resolver-wlzmw"
Jul 28 01:25:58.818232 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.817280    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-system-cni-dir\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.818232 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.817314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-multus-socket-dir-parent\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.818232 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.817377    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b8193a25-2add-42b3-8815-022a163695fe-serviceca\") pod \"node-ca-2d5z9\" (UID: \"b8193a25-2add-42b3-8815-022a163695fe\") " pod="openshift-image-registry/node-ca-2d5z9"
Jul 28 01:25:58.818232 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.817416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j78fv\" (UniqueName: \"kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv\") pod \"network-check-target-zzc9f\" (UID: \"e5ba3944-6409-4af9-9bff-8eb91a65755a\") " pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:25:58.818232 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.817460    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-modprobe-d\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.818705 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.818303    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k7tk6\""
Jul 28 01:25:58.822390 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.821065    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 01:25:58.822390 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.821536    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 01:25:58.822390 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.821569    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 01:25:58.822390 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.821882    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 01:25:58.822390 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.821913    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 01:25:58.823923 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.822754    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:58.823923 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.822972    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hqvhw\""
Jul 28 01:25:58.823923 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.823290    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 01:25:58.823923 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.823406    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 01:25:58.823923 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.823556    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 01:25:58.828177 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.828158    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 01:25:58.850380 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.850349    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mxq6k"
Jul 28 01:25:58.860000 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.859966    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mxq6k"
Jul 28 01:25:58.905843 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.905814    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 01:25:58.917680 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.917646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-hostroot\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.917680 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.917690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wcxc\" (UniqueName: \"kubernetes.io/projected/b8193a25-2add-42b3-8815-022a163695fe-kube-api-access-4wcxc\") pod \"node-ca-2d5z9\" (UID: \"b8193a25-2add-42b3-8815-022a163695fe\") " pod="openshift-image-registry/node-ca-2d5z9"
Jul 28 01:25:58.917912 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.917761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-hostroot\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.917912 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.917830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret\") pod \"global-pull-secret-syncer-wpcdj\" (UID: \"cd1a9e35-4116-41f7-b275-9da51eb2549a\") " pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:25:58.917912 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.917863    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-ovnkube-config\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.918067 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.917957    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-host-run-k8s-cni-cncf-io\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.918067 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:58.917974    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:58.918067 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.917984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-sysctl-conf\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.918067 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918001    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-host-run-k8s-cni-cncf-io\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.918067 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-run-ovn\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.918067 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:58.918067    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret podName:cd1a9e35-4116-41f7-b275-9da51eb2549a nodeName:}" failed. No retries permitted until 2026-07-28 01:25:59.418023426 +0000 UTC m=+2.995481021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret") pod "global-pull-secret-syncer-wpcdj" (UID: "cd1a9e35-4116-41f7-b275-9da51eb2549a") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:58.918381 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fbts\" (UniqueName: \"kubernetes.io/projected/ddcb15a3-037d-45a0-8787-12585c0d2465-kube-api-access-4fbts\") pod \"network-metrics-daemon-dx5sm\" (UID: \"ddcb15a3-037d-45a0-8787-12585c0d2465\") " pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:25:58.918381 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/640c683a-7ed6-4686-a0de-f6c631e1ab3a-hosts-file\") pod \"node-resolver-wlzmw\" (UID: \"640c683a-7ed6-4686-a0de-f6c631e1ab3a\") " pod="openshift-dns/node-resolver-wlzmw"
Jul 28 01:25:58.918381 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/05bfa172-3907-43d0-a248-b4c74dd7620d-socket-dir\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.918381 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918172    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-sysctl-conf\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.918381 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/05bfa172-3907-43d0-a248-b4c74dd7620d-etc-selinux\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.918381 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd1a9e35-4116-41f7-b275-9da51eb2549a-dbus\") pod \"global-pull-secret-syncer-wpcdj\" (UID: \"cd1a9e35-4116-41f7-b275-9da51eb2549a\") " pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:25:58.918381 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-kubernetes\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.918381 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918278    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/640c683a-7ed6-4686-a0de-f6c631e1ab3a-hosts-file\") pod \"node-resolver-wlzmw\" (UID: \"640c683a-7ed6-4686-a0de-f6c631e1ab3a\") " pod="openshift-dns/node-resolver-wlzmw"
Jul 28 01:25:58.918381 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-node-log\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.918381 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.918381 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65e2f293-2ed4-4ff1-aba1-ca582bde2904-cni-binary-copy\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.918883 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-host-var-lib-cni-multus\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.918883 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918399    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/05bfa172-3907-43d0-a248-b4c74dd7620d-etc-selinux\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.918883 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/05bfa172-3907-43d0-a248-b4c74dd7620d-device-dir\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.918883 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-sysctl-d\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.918883 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-kubelet\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.918883 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918488    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-run-netns\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.918883 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/05bfa172-3907-43d0-a248-b4c74dd7620d-socket-dir\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.918883 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918559    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-kubernetes\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.918883 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918568    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/05bfa172-3907-43d0-a248-b4c74dd7620d-device-dir\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.918883 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918574    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd1a9e35-4116-41f7-b275-9da51eb2549a-dbus\") pod \"global-pull-secret-syncer-wpcdj\" (UID: \"cd1a9e35-4116-41f7-b275-9da51eb2549a\") " pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:25:58.918883 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-sysconfig\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.918883 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918662    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-cni-bin\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.918883 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918690    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-host-var-lib-cni-multus\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.918883 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918700    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-sysconfig\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.918883 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918689    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-cnibin\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:58.918883 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918764    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/65e2f293-2ed4-4ff1-aba1-ca582bde2904-multus-daemon-config\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.918883 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918815    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-sysctl-d\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.919689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/05bfa172-3907-43d0-a248-b4c74dd7620d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.919689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/05bfa172-3907-43d0-a248-b4c74dd7620d-sys-fs\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.919689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918882    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf7mk\" (UniqueName: \"kubernetes.io/projected/05bfa172-3907-43d0-a248-b4c74dd7620d-kube-api-access-bf7mk\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.919689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-tmp\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.919689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-env-overrides\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.919689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-system-cni-dir\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:58.919689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/05bfa172-3907-43d0-a248-b4c74dd7620d-sys-fs\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.919689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.918982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:58.919689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-slash\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.919689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919024    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/05bfa172-3907-43d0-a248-b4c74dd7620d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.919689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qllg6\" (UniqueName: \"kubernetes.io/projected/5f007227-cbc6-4d32-a2c8-4ea6e62d606a-kube-api-access-qllg6\") pod \"iptables-alerter-qnv59\" (UID: \"5f007227-cbc6-4d32-a2c8-4ea6e62d606a\") " pod="openshift-network-operator/iptables-alerter-qnv59"
Jul 28 01:25:58.919689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919075    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65e2f293-2ed4-4ff1-aba1-ca582bde2904-cni-binary-copy\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.919689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/069b6671-6207-471b-a09c-26a0293ba1ba-agent-certs\") pod \"konnectivity-agent-c9ntv\" (UID: \"069b6671-6207-471b-a09c-26a0293ba1ba\") " pod="kube-system/konnectivity-agent-c9ntv"
Jul 28 01:25:58.919689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/640c683a-7ed6-4686-a0de-f6c631e1ab3a-tmp-dir\") pod \"node-resolver-wlzmw\" (UID: \"640c683a-7ed6-4686-a0de-f6c631e1ab3a\") " pod="openshift-dns/node-resolver-wlzmw"
Jul 28 01:25:58.919689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b8193a25-2add-42b3-8815-022a163695fe-serviceca\") pod \"node-ca-2d5z9\" (UID: \"b8193a25-2add-42b3-8815-022a163695fe\") " pod="openshift-image-registry/node-ca-2d5z9"
Jul 28 01:25:58.919689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/65e2f293-2ed4-4ff1-aba1-ca582bde2904-multus-daemon-config\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.919689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j78fv\" (UniqueName: \"kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv\") pod \"network-check-target-zzc9f\" (UID: \"e5ba3944-6409-4af9-9bff-8eb91a65755a\") " pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:25:58.920689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919322    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 01:25:58.920689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/640c683a-7ed6-4686-a0de-f6c631e1ab3a-tmp-dir\") pod \"node-resolver-wlzmw\" (UID: \"640c683a-7ed6-4686-a0de-f6c631e1ab3a\") " pod="openshift-dns/node-resolver-wlzmw"
Jul 28 01:25:58.920689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-modprobe-d\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.920689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919605    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b8193a25-2add-42b3-8815-022a163695fe-serviceca\") pod \"node-ca-2d5z9\" (UID: \"b8193a25-2add-42b3-8815-022a163695fe\") " pod="openshift-image-registry/node-ca-2d5z9"
Jul 28 01:25:58.920689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-systemd\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.920689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919636    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-modprobe-d\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.920689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-var-lib-openvswitch\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.920689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-systemd\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.920689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z429\" (UniqueName: \"kubernetes.io/projected/640c683a-7ed6-4686-a0de-f6c631e1ab3a-kube-api-access-5z429\") pod \"node-resolver-wlzmw\" (UID: \"640c683a-7ed6-4686-a0de-f6c631e1ab3a\") " pod="openshift-dns/node-resolver-wlzmw"
Jul 28 01:25:58.920689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-host-var-lib-cni-bin\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.920689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-host-var-lib-kubelet\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.920689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/05bfa172-3907-43d0-a248-b4c74dd7620d-registration-dir\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.920689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919799    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-host-var-lib-kubelet\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.920689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919826    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-host-var-lib-cni-bin\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.920689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919854    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/05bfa172-3907-43d0-a248-b4c74dd7620d-registration-dir\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.920689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-lib-modules\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.920689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-tuned\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.920689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmmbx\" (UniqueName: \"kubernetes.io/projected/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-kube-api-access-tmmbx\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.921583 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.919984    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-lib-modules\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.921583 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920004    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-run-systemd\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.921583 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920032    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-host-run-multus-certs\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.921583 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2ddg\" (UniqueName: \"kubernetes.io/projected/65e2f293-2ed4-4ff1-aba1-ca582bde2904-kube-api-access-h2ddg\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.921583 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920123    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-host-run-multus-certs\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.921583 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-sys\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.921583 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920174    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd1a9e35-4116-41f7-b275-9da51eb2549a-kubelet-config\") pod \"global-pull-secret-syncer-wpcdj\" (UID: \"cd1a9e35-4116-41f7-b275-9da51eb2549a\") " pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:25:58.921583 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-host\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.921583 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8193a25-2add-42b3-8815-022a163695fe-host\") pod \"node-ca-2d5z9\" (UID: \"b8193a25-2add-42b3-8815-022a163695fe\") " pod="openshift-image-registry/node-ca-2d5z9"
Jul 28 01:25:58.921583 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-os-release\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.921583 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd1a9e35-4116-41f7-b275-9da51eb2549a-kubelet-config\") pod \"global-pull-secret-syncer-wpcdj\" (UID: \"cd1a9e35-4116-41f7-b275-9da51eb2549a\") " pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:25:58.921583 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-host\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.921583 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920347    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8193a25-2add-42b3-8815-022a163695fe-host\") pod \"node-ca-2d5z9\" (UID: \"b8193a25-2add-42b3-8815-022a163695fe\") " pod="openshift-image-registry/node-ca-2d5z9"
Jul 28 01:25:58.921583 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920289    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-systemd-units\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.921583 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-log-socket\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.921583 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-ovn-node-metrics-cert\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.921583 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-os-release\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.921583 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-os-release\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:58.922433 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920484    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-sys\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.922433 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5f007227-cbc6-4d32-a2c8-4ea6e62d606a-iptables-alerter-script\") pod \"iptables-alerter-qnv59\" (UID: \"5f007227-cbc6-4d32-a2c8-4ea6e62d606a\") " pod="openshift-network-operator/iptables-alerter-qnv59"
Jul 28 01:25:58.922433 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-run-openvswitch\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.922433 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920557    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:58.922433 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-multus-cni-dir\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.922433 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-multus-conf-dir\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.922433 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-etc-openvswitch\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.922433 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:58.922433 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920672    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-multus-cni-dir\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.922433 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920664    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-multus-conf-dir\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.922433 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f007227-cbc6-4d32-a2c8-4ea6e62d606a-host-slash\") pod \"iptables-alerter-qnv59\" (UID: \"5f007227-cbc6-4d32-a2c8-4ea6e62d606a\") " pod="openshift-network-operator/iptables-alerter-qnv59"
Jul 28 01:25:58.922433 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs\") pod \"network-metrics-daemon-dx5sm\" (UID: \"ddcb15a3-037d-45a0-8787-12585c0d2465\") " pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:25:58.922433 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-cnibin\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.922433 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920789    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-cnibin\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.922433 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-host-run-netns\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.922433 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:58.920820    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:58.922433 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-etc-kubernetes\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.922433 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920853    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/069b6671-6207-471b-a09c-26a0293ba1ba-konnectivity-ca\") pod \"konnectivity-agent-c9ntv\" (UID: \"069b6671-6207-471b-a09c-26a0293ba1ba\") " pod="kube-system/konnectivity-agent-c9ntv"
Jul 28 01:25:58.923284 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:58.920881    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs podName:ddcb15a3-037d-45a0-8787-12585c0d2465 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:59.420864854 +0000 UTC m=+2.998322449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs") pod "network-metrics-daemon-dx5sm" (UID: "ddcb15a3-037d-45a0-8787-12585c0d2465") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:58.923284 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920901    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.923284 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920911    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-host-run-netns\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.923284 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920929    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svhvq\" (UniqueName: \"kubernetes.io/projected/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-kube-api-access-svhvq\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.923284 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920971    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-etc-kubernetes\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.923284 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.920996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-run\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.923284 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.921028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-var-lib-kubelet\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.923284 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.921055    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-system-cni-dir\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.923284 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.921073    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-run\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.923284 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.921081    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-multus-socket-dir-parent\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.923284 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.921110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-cni-netd\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.923284 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.921155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-ovnkube-script-lib\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:58.923284 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.921180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:58.923284 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.921194    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-system-cni-dir\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.923284 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.921206    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkntw\" (UniqueName: \"kubernetes.io/projected/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-kube-api-access-lkntw\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:58.923284 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.921238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-var-lib-kubelet\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.924075 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.921305    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/65e2f293-2ed4-4ff1-aba1-ca582bde2904-multus-socket-dir-parent\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.924075 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.921380    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/069b6671-6207-471b-a09c-26a0293ba1ba-konnectivity-ca\") pod \"konnectivity-agent-c9ntv\" (UID: \"069b6671-6207-471b-a09c-26a0293ba1ba\") " pod="kube-system/konnectivity-agent-c9ntv"
Jul 28 01:25:58.924075 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.922963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-tmp\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.924075 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.923417    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/069b6671-6207-471b-a09c-26a0293ba1ba-agent-certs\") pod \"konnectivity-agent-c9ntv\" (UID: \"069b6671-6207-471b-a09c-26a0293ba1ba\") " pod="kube-system/konnectivity-agent-c9ntv"
Jul 28 01:25:58.924492 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.924315    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-etc-tuned\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.928359 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.928295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wcxc\" (UniqueName: \"kubernetes.io/projected/b8193a25-2add-42b3-8815-022a163695fe-kube-api-access-4wcxc\") pod \"node-ca-2d5z9\" (UID: \"b8193a25-2add-42b3-8815-022a163695fe\") " pod="openshift-image-registry/node-ca-2d5z9"
Jul 28 01:25:58.929881 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:58.929341    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:58.929881 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:58.929364    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:58.929881 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:58.929378    2578 projected.go:196] Error preparing data for projected volume kube-api-access-j78fv for pod openshift-network-diagnostics/network-check-target-zzc9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:58.929881 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:58.929441    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv podName:e5ba3944-6409-4af9-9bff-8eb91a65755a nodeName:}" failed. No retries permitted until 2026-07-28 01:25:59.429423763 +0000 UTC m=+3.006881359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j78fv" (UniqueName: "kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv") pod "network-check-target-zzc9f" (UID: "e5ba3944-6409-4af9-9bff-8eb91a65755a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:58.930266 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.930214    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fbts\" (UniqueName: \"kubernetes.io/projected/ddcb15a3-037d-45a0-8787-12585c0d2465-kube-api-access-4fbts\") pod \"network-metrics-daemon-dx5sm\" (UID: \"ddcb15a3-037d-45a0-8787-12585c0d2465\") " pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:25:58.931446 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.931414    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2ddg\" (UniqueName: \"kubernetes.io/projected/65e2f293-2ed4-4ff1-aba1-ca582bde2904-kube-api-access-h2ddg\") pod \"multus-qqq2z\" (UID: \"65e2f293-2ed4-4ff1-aba1-ca582bde2904\") " pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:58.932111 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.932062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmmbx\" (UniqueName: \"kubernetes.io/projected/5cdf4c08-dce5-43fd-b3b8-c651c49b99c3-kube-api-access-tmmbx\") pod \"tuned-bh7l6\" (UID: \"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:58.933574 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.933522    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z429\" (UniqueName: \"kubernetes.io/projected/640c683a-7ed6-4686-a0de-f6c631e1ab3a-kube-api-access-5z429\") pod \"node-resolver-wlzmw\" (UID: \"640c683a-7ed6-4686-a0de-f6c631e1ab3a\") " pod="openshift-dns/node-resolver-wlzmw"
Jul 28 01:25:58.933764 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.933719    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf7mk\" (UniqueName: \"kubernetes.io/projected/05bfa172-3907-43d0-a248-b4c74dd7620d-kube-api-access-bf7mk\") pod \"aws-ebs-csi-driver-node-bmx5t\" (UID: \"05bfa172-3907-43d0-a248-b4c74dd7620d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:58.984806 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.984746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal" event={"ID":"4fa6e8ab638aa3b95e5e62c2f5cb6258","Type":"ContainerStarted","Data":"580847dd2337b1aa40c8c8744fd59ab84d1fd9e1a071f0d998863521b79d9511"}
Jul 28 01:25:58.986153 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:58.986125    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal" event={"ID":"216527ecc9658544f9bc7b3b7800c13c","Type":"ContainerStarted","Data":"6ea7135b723ff52ff21e3f5f2f630a5f88be12dce600c75e8cfc540707e43100"}
Jul 28 01:25:59.022256 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-run-netns\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.022256 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-cni-bin\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.022477 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022280    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-cnibin\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:59.022477 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-env-overrides\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.022477 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-system-cni-dir\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:59.022477 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022355    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-run-netns\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.022477 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022361    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-cni-bin\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.022477 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022403    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-system-cni-dir\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:59.022477 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:59.022477 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-slash\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.022477 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qllg6\" (UniqueName: \"kubernetes.io/projected/5f007227-cbc6-4d32-a2c8-4ea6e62d606a-kube-api-access-qllg6\") pod \"iptables-alerter-qnv59\" (UID: \"5f007227-cbc6-4d32-a2c8-4ea6e62d606a\") " pod="openshift-network-operator/iptables-alerter-qnv59"
Jul 28 01:25:59.022903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-var-lib-openvswitch\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.022903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-run-systemd\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.022903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-systemd-units\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.022903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-log-socket\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.022903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022618    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-ovn-node-metrics-cert\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.022903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022619    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-var-lib-openvswitch\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.022903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-os-release\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:59.022903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022632    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-run-systemd\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.022903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5f007227-cbc6-4d32-a2c8-4ea6e62d606a-iptables-alerter-script\") pod \"iptables-alerter-qnv59\" (UID: \"5f007227-cbc6-4d32-a2c8-4ea6e62d606a\") " pod="openshift-network-operator/iptables-alerter-qnv59"
Jul 28 01:25:59.022903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022682    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-cnibin\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:59.022903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-run-openvswitch\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.022903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022695    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-log-socket\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.022903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022714    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:59.022903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022725    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-slash\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.022903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022733    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-systemd-units\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.022903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-etc-openvswitch\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.022903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:59.023735 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f007227-cbc6-4d32-a2c8-4ea6e62d606a-host-slash\") pod \"iptables-alerter-qnv59\" (UID: \"5f007227-cbc6-4d32-a2c8-4ea6e62d606a\") " pod="openshift-network-operator/iptables-alerter-qnv59"
Jul 28 01:25:59.023735 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022837    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-run-openvswitch\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.023735 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022843    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-os-release\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:59.023735 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022895    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f007227-cbc6-4d32-a2c8-4ea6e62d606a-host-slash\") pod \"iptables-alerter-qnv59\" (UID: \"5f007227-cbc6-4d32-a2c8-4ea6e62d606a\") " pod="openshift-network-operator/iptables-alerter-qnv59"
Jul 28 01:25:59.023735 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.023735 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.023735 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022929    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:59.023735 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022938    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-etc-openvswitch\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.023735 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022947    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svhvq\" (UniqueName: \"kubernetes.io/projected/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-kube-api-access-svhvq\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.023735 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-cni-netd\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.023735 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.022999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-ovnkube-script-lib\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.023735 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.023023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:59.023735 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.023048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkntw\" (UniqueName: \"kubernetes.io/projected/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-kube-api-access-lkntw\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:59.023735 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.023087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-ovnkube-config\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.023735 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.023115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-run-ovn\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.023735 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.023148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-node-log\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.023735 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.023171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.024541 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.023201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-kubelet\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.024541 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.023209    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:59.024541 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.023270    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-kubelet\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.024541 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.023278    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:59.024541 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.023284    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5f007227-cbc6-4d32-a2c8-4ea6e62d606a-iptables-alerter-script\") pod \"iptables-alerter-qnv59\" (UID: \"5f007227-cbc6-4d32-a2c8-4ea6e62d606a\") " pod="openshift-network-operator/iptables-alerter-qnv59"
Jul 28 01:25:59.024541 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.023281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-run-ovn\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.024541 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.023270    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:59.024541 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.023380    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.024541 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.023405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-node-log\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.024541 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.023420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-host-cni-netd\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.024541 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.023536    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-env-overrides\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.024541 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.023773    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-ovnkube-config\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.024541 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.023804    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-ovnkube-script-lib\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.026232 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.026207    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-ovn-node-metrics-cert\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.034220 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.034132    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkntw\" (UniqueName: \"kubernetes.io/projected/69bc94d5-8275-4e49-b94f-03ef28a1e2b2-kube-api-access-lkntw\") pod \"multus-additional-cni-plugins-gwwvx\" (UID: \"69bc94d5-8275-4e49-b94f-03ef28a1e2b2\") " pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:59.035097 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.035051    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qllg6\" (UniqueName: \"kubernetes.io/projected/5f007227-cbc6-4d32-a2c8-4ea6e62d606a-kube-api-access-qllg6\") pod \"iptables-alerter-qnv59\" (UID: \"5f007227-cbc6-4d32-a2c8-4ea6e62d606a\") " pod="openshift-network-operator/iptables-alerter-qnv59"
Jul 28 01:25:59.035320 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.035300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svhvq\" (UniqueName: \"kubernetes.io/projected/e12a776b-5a0c-40f7-a2d1-221f8dcb27fc-kube-api-access-svhvq\") pod \"ovnkube-node-l5mxg\" (UID: \"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.107537 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.107507    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2d5z9"
Jul 28 01:25:59.116738 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.116709    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t"
Jul 28 01:25:59.127403 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.127374    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wlzmw"
Jul 28 01:25:59.129534 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:59.129506    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05bfa172_3907_43d0_a248_b4c74dd7620d.slice/crio-141b6abd3bcf247b531cbf504944c529c2c2a296e17c64fd366b1800d9191384 WatchSource:0}: Error finding container 141b6abd3bcf247b531cbf504944c529c2c2a296e17c64fd366b1800d9191384: Status 404 returned error can't find the container with id 141b6abd3bcf247b531cbf504944c529c2c2a296e17c64fd366b1800d9191384
Jul 28 01:25:59.131275 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:59.130968    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8193a25_2add_42b3_8815_022a163695fe.slice/crio-5f6ecd36103419da32ce0f295780cf52dd73bba836df91a4b6b430b0bfef55c1 WatchSource:0}: Error finding container 5f6ecd36103419da32ce0f295780cf52dd73bba836df91a4b6b430b0bfef55c1: Status 404 returned error can't find the container with id 5f6ecd36103419da32ce0f295780cf52dd73bba836df91a4b6b430b0bfef55c1
Jul 28 01:25:59.131758 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.131590    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qqq2z"
Jul 28 01:25:59.139430 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.139411    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c9ntv"
Jul 28 01:25:59.140878 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:59.140858    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640c683a_7ed6_4686_a0de_f6c631e1ab3a.slice/crio-4adda57d92a325090dccd525f5d0c4c5eaac7dd72c7e515e43d08b005c29828e WatchSource:0}: Error finding container 4adda57d92a325090dccd525f5d0c4c5eaac7dd72c7e515e43d08b005c29828e: Status 404 returned error can't find the container with id 4adda57d92a325090dccd525f5d0c4c5eaac7dd72c7e515e43d08b005c29828e
Jul 28 01:25:59.141591 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:59.141548    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e2f293_2ed4_4ff1_aba1_ca582bde2904.slice/crio-2e1ac09f59aa18c37692971d260b01552d865f67f336a57e2a6d76d1cbe7d7c6 WatchSource:0}: Error finding container 2e1ac09f59aa18c37692971d260b01552d865f67f336a57e2a6d76d1cbe7d7c6: Status 404 returned error can't find the container with id 2e1ac09f59aa18c37692971d260b01552d865f67f336a57e2a6d76d1cbe7d7c6
Jul 28 01:25:59.148092 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.148069    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bh7l6"
Jul 28 01:25:59.150106 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:59.150078    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069b6671_6207_471b_a09c_26a0293ba1ba.slice/crio-8f5f1fa067de2acd4d5756b6208fd8db0eee34864099266795054dc6f63c12da WatchSource:0}: Error finding container 8f5f1fa067de2acd4d5756b6208fd8db0eee34864099266795054dc6f63c12da: Status 404 returned error can't find the container with id 8f5f1fa067de2acd4d5756b6208fd8db0eee34864099266795054dc6f63c12da
Jul 28 01:25:59.154727 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.154706    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gwwvx"
Jul 28 01:25:59.156846 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:59.156822    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cdf4c08_dce5_43fd_b3b8_c651c49b99c3.slice/crio-a08fdc28627a5f9172e973a1113079873b30b1c511ded78922c71b3f49a3182d WatchSource:0}: Error finding container a08fdc28627a5f9172e973a1113079873b30b1c511ded78922c71b3f49a3182d: Status 404 returned error can't find the container with id a08fdc28627a5f9172e973a1113079873b30b1c511ded78922c71b3f49a3182d
Jul 28 01:25:59.163661 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.163640    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qnv59"
Jul 28 01:25:59.164236 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:59.164213    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69bc94d5_8275_4e49_b94f_03ef28a1e2b2.slice/crio-47a5e2e156cd118406afbc24e8db3ee390467ed2710b5b7bb5faa6995a5d32bc WatchSource:0}: Error finding container 47a5e2e156cd118406afbc24e8db3ee390467ed2710b5b7bb5faa6995a5d32bc: Status 404 returned error can't find the container with id 47a5e2e156cd118406afbc24e8db3ee390467ed2710b5b7bb5faa6995a5d32bc
Jul 28 01:25:59.169650 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.169632    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:25:59.174834 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:25:59.174609    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f007227_cbc6_4d32_a2c8_4ea6e62d606a.slice/crio-345c1765b712d36561839411065cf04c393ae7515afcd4981059593e2e8ff9ea WatchSource:0}: Error finding container 345c1765b712d36561839411065cf04c393ae7515afcd4981059593e2e8ff9ea: Status 404 returned error can't find the container with id 345c1765b712d36561839411065cf04c393ae7515afcd4981059593e2e8ff9ea
Jul 28 01:25:59.426394 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.426360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs\") pod \"network-metrics-daemon-dx5sm\" (UID: \"ddcb15a3-037d-45a0-8787-12585c0d2465\") " pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:25:59.426580 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.426426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret\") pod \"global-pull-secret-syncer-wpcdj\" (UID: \"cd1a9e35-4116-41f7-b275-9da51eb2549a\") " pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:25:59.426580 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:59.426467    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:59.426580 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:59.426544    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs podName:ddcb15a3-037d-45a0-8787-12585c0d2465 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:00.426523664 +0000 UTC m=+4.003981272 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs") pod "network-metrics-daemon-dx5sm" (UID: "ddcb15a3-037d-45a0-8787-12585c0d2465") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:59.426751 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:59.426617    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:59.426751 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:59.426679    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret podName:cd1a9e35-4116-41f7-b275-9da51eb2549a nodeName:}" failed. No retries permitted until 2026-07-28 01:26:00.426660374 +0000 UTC m=+4.004117971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret") pod "global-pull-secret-syncer-wpcdj" (UID: "cd1a9e35-4116-41f7-b275-9da51eb2549a") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:59.526819 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.526786    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j78fv\" (UniqueName: \"kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv\") pod \"network-check-target-zzc9f\" (UID: \"e5ba3944-6409-4af9-9bff-8eb91a65755a\") " pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:25:59.526970 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:59.526911    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:59.526970 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:59.526927    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:59.526970 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:59.526936    2578 projected.go:196] Error preparing data for projected volume kube-api-access-j78fv for pod openshift-network-diagnostics/network-check-target-zzc9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:59.527084 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:25:59.526983    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv podName:e5ba3944-6409-4af9-9bff-8eb91a65755a nodeName:}" failed. No retries permitted until 2026-07-28 01:26:00.52696991 +0000 UTC m=+4.104427506 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j78fv" (UniqueName: "kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv") pod "network-check-target-zzc9f" (UID: "e5ba3944-6409-4af9-9bff-8eb91a65755a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:59.862633 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.862577    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 01:20:58 +0000 UTC" deadline="2028-04-01 22:23:14.84345483 +0000 UTC"
Jul 28 01:25:59.862633 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.862631    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14732h57m14.980828656s"
Jul 28 01:25:59.989844 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.989806    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qqq2z" event={"ID":"65e2f293-2ed4-4ff1-aba1-ca582bde2904","Type":"ContainerStarted","Data":"2e1ac09f59aa18c37692971d260b01552d865f67f336a57e2a6d76d1cbe7d7c6"}
Jul 28 01:25:59.994557 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.994525    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t" event={"ID":"05bfa172-3907-43d0-a248-b4c74dd7620d","Type":"ContainerStarted","Data":"141b6abd3bcf247b531cbf504944c529c2c2a296e17c64fd366b1800d9191384"}
Jul 28 01:25:59.996253 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.996225    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bh7l6" event={"ID":"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3","Type":"ContainerStarted","Data":"a08fdc28627a5f9172e973a1113079873b30b1c511ded78922c71b3f49a3182d"}
Jul 28 01:25:59.998918 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:25:59.998890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c9ntv" event={"ID":"069b6671-6207-471b-a09c-26a0293ba1ba","Type":"ContainerStarted","Data":"8f5f1fa067de2acd4d5756b6208fd8db0eee34864099266795054dc6f63c12da"}
Jul 28 01:26:00.001262 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:00.001236    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wlzmw" event={"ID":"640c683a-7ed6-4686-a0de-f6c631e1ab3a","Type":"ContainerStarted","Data":"4adda57d92a325090dccd525f5d0c4c5eaac7dd72c7e515e43d08b005c29828e"}
Jul 28 01:26:00.005767 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:00.005714    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2d5z9" event={"ID":"b8193a25-2add-42b3-8815-022a163695fe","Type":"ContainerStarted","Data":"5f6ecd36103419da32ce0f295780cf52dd73bba836df91a4b6b430b0bfef55c1"}
Jul 28 01:26:00.008916 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:00.008886    2578 generic.go:356] "Generic (PLEG): container finished" podID="4fa6e8ab638aa3b95e5e62c2f5cb6258" containerID="fa95547e93e6f2a707026e66089556b24778c18040f41cd8b2c9fefedf53375d" exitCode=0
Jul 28 01:26:00.009004 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:00.008970    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal" event={"ID":"4fa6e8ab638aa3b95e5e62c2f5cb6258","Type":"ContainerDied","Data":"fa95547e93e6f2a707026e66089556b24778c18040f41cd8b2c9fefedf53375d"}
Jul 28 01:26:00.014435 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:00.014066    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg" event={"ID":"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc","Type":"ContainerStarted","Data":"cf4f57f2506f3d6b2afc8bc536e6b103c021e9f0acc5cc262018b5472167c40a"}
Jul 28 01:26:00.016483 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:00.016461    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qnv59" event={"ID":"5f007227-cbc6-4d32-a2c8-4ea6e62d606a","Type":"ContainerStarted","Data":"345c1765b712d36561839411065cf04c393ae7515afcd4981059593e2e8ff9ea"}
Jul 28 01:26:00.018777 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:00.018733    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwwvx" event={"ID":"69bc94d5-8275-4e49-b94f-03ef28a1e2b2","Type":"ContainerStarted","Data":"47a5e2e156cd118406afbc24e8db3ee390467ed2710b5b7bb5faa6995a5d32bc"}
Jul 28 01:26:00.435015 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:00.434230    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs\") pod \"network-metrics-daemon-dx5sm\" (UID: \"ddcb15a3-037d-45a0-8787-12585c0d2465\") " pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:00.435015 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:00.434287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret\") pod \"global-pull-secret-syncer-wpcdj\" (UID: \"cd1a9e35-4116-41f7-b275-9da51eb2549a\") " pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:00.435015 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:00.434434    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:00.435015 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:00.434500    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret podName:cd1a9e35-4116-41f7-b275-9da51eb2549a nodeName:}" failed. No retries permitted until 2026-07-28 01:26:02.434482396 +0000 UTC m=+6.011939997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret") pod "global-pull-secret-syncer-wpcdj" (UID: "cd1a9e35-4116-41f7-b275-9da51eb2549a") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:00.435015 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:00.434877    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:00.435015 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:00.434927    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs podName:ddcb15a3-037d-45a0-8787-12585c0d2465 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:02.43491182 +0000 UTC m=+6.012369420 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs") pod "network-metrics-daemon-dx5sm" (UID: "ddcb15a3-037d-45a0-8787-12585c0d2465") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:00.535552 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:00.534939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j78fv\" (UniqueName: \"kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv\") pod \"network-check-target-zzc9f\" (UID: \"e5ba3944-6409-4af9-9bff-8eb91a65755a\") " pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:00.535552 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:00.535111    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:00.535552 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:00.535130    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:00.535552 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:00.535142    2578 projected.go:196] Error preparing data for projected volume kube-api-access-j78fv for pod openshift-network-diagnostics/network-check-target-zzc9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:00.535552 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:00.535200    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv podName:e5ba3944-6409-4af9-9bff-8eb91a65755a nodeName:}" failed. No retries permitted until 2026-07-28 01:26:02.535181739 +0000 UTC m=+6.112639339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j78fv" (UniqueName: "kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv") pod "network-check-target-zzc9f" (UID: "e5ba3944-6409-4af9-9bff-8eb91a65755a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:00.863156 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:00.863081    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 01:20:58 +0000 UTC" deadline="2028-04-23 18:21:57.691277836 +0000 UTC"
Jul 28 01:26:00.863156 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:00.863155    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15256h55m56.828127456s"
Jul 28 01:26:00.983016 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:00.982181    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:00.983016 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:00.982312    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpcdj" podUID="cd1a9e35-4116-41f7-b275-9da51eb2549a"
Jul 28 01:26:00.983016 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:00.982415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:00.983016 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:00.982487    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzc9f" podUID="e5ba3944-6409-4af9-9bff-8eb91a65755a"
Jul 28 01:26:00.983016 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:00.982552    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:00.983016 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:00.982637    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dx5sm" podUID="ddcb15a3-037d-45a0-8787-12585c0d2465"
Jul 28 01:26:01.036773 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:01.036735    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal" event={"ID":"216527ecc9658544f9bc7b3b7800c13c","Type":"ContainerStarted","Data":"133dea09969f2295af9eb1fd6895e3e0438f9a94c49238de13ea49c52c1a7e15"}
Jul 28 01:26:01.044786 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:01.044756    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal" event={"ID":"4fa6e8ab638aa3b95e5e62c2f5cb6258","Type":"ContainerStarted","Data":"39b15317edaf7ffd596dbe5be1ca1cc2a9b724c97419096182b7617fe9d2356c"}
Jul 28 01:26:01.072679 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:01.072618    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-81.ec2.internal" podStartSLOduration=3.07259804 podStartE2EDuration="3.07259804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:01.055130093 +0000 UTC m=+4.632587710" watchObservedRunningTime="2026-07-28 01:26:01.07259804 +0000 UTC m=+4.650055658"
Jul 28 01:26:02.452379 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:02.451612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs\") pod \"network-metrics-daemon-dx5sm\" (UID: \"ddcb15a3-037d-45a0-8787-12585c0d2465\") " pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:02.452379 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:02.451674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret\") pod \"global-pull-secret-syncer-wpcdj\" (UID: \"cd1a9e35-4116-41f7-b275-9da51eb2549a\") " pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:02.452379 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:02.451840    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:02.452379 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:02.451902    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret podName:cd1a9e35-4116-41f7-b275-9da51eb2549a nodeName:}" failed. No retries permitted until 2026-07-28 01:26:06.451883442 +0000 UTC m=+10.029341038 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret") pod "global-pull-secret-syncer-wpcdj" (UID: "cd1a9e35-4116-41f7-b275-9da51eb2549a") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:02.452379 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:02.452312    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:02.453173 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:02.453140    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs podName:ddcb15a3-037d-45a0-8787-12585c0d2465 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:06.45311759 +0000 UTC m=+10.030575199 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs") pod "network-metrics-daemon-dx5sm" (UID: "ddcb15a3-037d-45a0-8787-12585c0d2465") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:02.552258 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:02.552216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j78fv\" (UniqueName: \"kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv\") pod \"network-check-target-zzc9f\" (UID: \"e5ba3944-6409-4af9-9bff-8eb91a65755a\") " pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:02.552491 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:02.552473    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:02.552553 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:02.552498    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:02.552553 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:02.552512    2578 projected.go:196] Error preparing data for projected volume kube-api-access-j78fv for pod openshift-network-diagnostics/network-check-target-zzc9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:02.552640 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:02.552561    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv podName:e5ba3944-6409-4af9-9bff-8eb91a65755a nodeName:}" failed. No retries permitted until 2026-07-28 01:26:06.552547373 +0000 UTC m=+10.130004968 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j78fv" (UniqueName: "kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv") pod "network-check-target-zzc9f" (UID: "e5ba3944-6409-4af9-9bff-8eb91a65755a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:02.981287 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:02.980488    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:02.981287 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:02.980637    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpcdj" podUID="cd1a9e35-4116-41f7-b275-9da51eb2549a"
Jul 28 01:26:02.981287 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:02.981014    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:02.981287 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:02.981107    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzc9f" podUID="e5ba3944-6409-4af9-9bff-8eb91a65755a"
Jul 28 01:26:02.981287 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:02.981162    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:02.981287 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:02.981244    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dx5sm" podUID="ddcb15a3-037d-45a0-8787-12585c0d2465"
Jul 28 01:26:04.980563 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:04.980449    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:04.980563 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:04.980474    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:04.981071 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:04.980597    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpcdj" podUID="cd1a9e35-4116-41f7-b275-9da51eb2549a"
Jul 28 01:26:04.981071 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:04.980658    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzc9f" podUID="e5ba3944-6409-4af9-9bff-8eb91a65755a"
Jul 28 01:26:04.981071 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:04.980720    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:04.981071 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:04.980800    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dx5sm" podUID="ddcb15a3-037d-45a0-8787-12585c0d2465"
Jul 28 01:26:06.487725 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:06.487694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs\") pod \"network-metrics-daemon-dx5sm\" (UID: \"ddcb15a3-037d-45a0-8787-12585c0d2465\") " pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:06.487725 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:06.487742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret\") pod \"global-pull-secret-syncer-wpcdj\" (UID: \"cd1a9e35-4116-41f7-b275-9da51eb2549a\") " pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:06.488243 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:06.487860    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:06.488243 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:06.487903    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret podName:cd1a9e35-4116-41f7-b275-9da51eb2549a nodeName:}" failed. No retries permitted until 2026-07-28 01:26:14.487890526 +0000 UTC m=+18.065348121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret") pod "global-pull-secret-syncer-wpcdj" (UID: "cd1a9e35-4116-41f7-b275-9da51eb2549a") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:06.488243 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:06.488182    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:06.488243 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:06.488213    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs podName:ddcb15a3-037d-45a0-8787-12585c0d2465 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:14.488203224 +0000 UTC m=+18.065660819 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs") pod "network-metrics-daemon-dx5sm" (UID: "ddcb15a3-037d-45a0-8787-12585c0d2465") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:06.588955 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:06.588425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j78fv\" (UniqueName: \"kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv\") pod \"network-check-target-zzc9f\" (UID: \"e5ba3944-6409-4af9-9bff-8eb91a65755a\") " pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:06.588955 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:06.588576    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:06.588955 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:06.588591    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:06.588955 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:06.588601    2578 projected.go:196] Error preparing data for projected volume kube-api-access-j78fv for pod openshift-network-diagnostics/network-check-target-zzc9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:06.588955 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:06.588668    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv podName:e5ba3944-6409-4af9-9bff-8eb91a65755a nodeName:}" failed. No retries permitted until 2026-07-28 01:26:14.588647007 +0000 UTC m=+18.166104608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j78fv" (UniqueName: "kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv") pod "network-check-target-zzc9f" (UID: "e5ba3944-6409-4af9-9bff-8eb91a65755a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:06.981960 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:06.981214    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:06.981960 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:06.981367    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dx5sm" podUID="ddcb15a3-037d-45a0-8787-12585c0d2465"
Jul 28 01:26:06.981960 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:06.981704    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:06.981960 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:06.981806    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpcdj" podUID="cd1a9e35-4116-41f7-b275-9da51eb2549a"
Jul 28 01:26:06.981960 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:06.981849    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:06.981960 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:06.981917    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzc9f" podUID="e5ba3944-6409-4af9-9bff-8eb91a65755a"
Jul 28 01:26:08.979966 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:08.979926    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:08.979966 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:08.979959    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:08.980505 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:08.979933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:08.980505 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:08.980085    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dx5sm" podUID="ddcb15a3-037d-45a0-8787-12585c0d2465"
Jul 28 01:26:08.980505 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:08.980195    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpcdj" podUID="cd1a9e35-4116-41f7-b275-9da51eb2549a"
Jul 28 01:26:08.980505 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:08.980294    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzc9f" podUID="e5ba3944-6409-4af9-9bff-8eb91a65755a"
Jul 28 01:26:10.979739 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:10.979706    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:10.980149 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:10.979706    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:10.980149 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:10.979834    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dx5sm" podUID="ddcb15a3-037d-45a0-8787-12585c0d2465"
Jul 28 01:26:10.980149 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:10.979885    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzc9f" podUID="e5ba3944-6409-4af9-9bff-8eb91a65755a"
Jul 28 01:26:10.980149 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:10.979706    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:10.980149 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:10.979983    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpcdj" podUID="cd1a9e35-4116-41f7-b275-9da51eb2549a"
Jul 28 01:26:12.980166 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:12.980128    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:12.980684 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:12.980128    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:12.980684 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:12.980254    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzc9f" podUID="e5ba3944-6409-4af9-9bff-8eb91a65755a"
Jul 28 01:26:12.980684 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:12.980128    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:12.980684 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:12.980382    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpcdj" podUID="cd1a9e35-4116-41f7-b275-9da51eb2549a"
Jul 28 01:26:12.980684 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:12.980491    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dx5sm" podUID="ddcb15a3-037d-45a0-8787-12585c0d2465"
Jul 28 01:26:14.543643 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:14.543599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret\") pod \"global-pull-secret-syncer-wpcdj\" (UID: \"cd1a9e35-4116-41f7-b275-9da51eb2549a\") " pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:14.544186 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:14.543709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs\") pod \"network-metrics-daemon-dx5sm\" (UID: \"ddcb15a3-037d-45a0-8787-12585c0d2465\") " pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:14.544186 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:14.543749    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:14.544186 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:14.543812    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:14.544186 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:14.543822    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret podName:cd1a9e35-4116-41f7-b275-9da51eb2549a nodeName:}" failed. No retries permitted until 2026-07-28 01:26:30.543800522 +0000 UTC m=+34.121258119 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret") pod "global-pull-secret-syncer-wpcdj" (UID: "cd1a9e35-4116-41f7-b275-9da51eb2549a") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:14.544186 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:14.543867    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs podName:ddcb15a3-037d-45a0-8787-12585c0d2465 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:30.543850324 +0000 UTC m=+34.121307919 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs") pod "network-metrics-daemon-dx5sm" (UID: "ddcb15a3-037d-45a0-8787-12585c0d2465") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:14.644639 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:14.644603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j78fv\" (UniqueName: \"kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv\") pod \"network-check-target-zzc9f\" (UID: \"e5ba3944-6409-4af9-9bff-8eb91a65755a\") " pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:14.644842 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:14.644818    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:14.644912 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:14.644852    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:14.644912 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:14.644866    2578 projected.go:196] Error preparing data for projected volume kube-api-access-j78fv for pod openshift-network-diagnostics/network-check-target-zzc9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:14.645007 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:14.644954    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv podName:e5ba3944-6409-4af9-9bff-8eb91a65755a nodeName:}" failed. No retries permitted until 2026-07-28 01:26:30.644933072 +0000 UTC m=+34.222390687 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j78fv" (UniqueName: "kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv") pod "network-check-target-zzc9f" (UID: "e5ba3944-6409-4af9-9bff-8eb91a65755a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:14.980288 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:14.980180    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:14.980288 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:14.980223    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:14.980553 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:14.980181    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:14.980553 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:14.980322    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzc9f" podUID="e5ba3944-6409-4af9-9bff-8eb91a65755a"
Jul 28 01:26:14.980553 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:14.980437    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dx5sm" podUID="ddcb15a3-037d-45a0-8787-12585c0d2465"
Jul 28 01:26:14.980553 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:14.980529    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpcdj" podUID="cd1a9e35-4116-41f7-b275-9da51eb2549a"
Jul 28 01:26:16.980387 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:16.980345    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:16.980845 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:16.980473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:16.980845 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:16.980482    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpcdj" podUID="cd1a9e35-4116-41f7-b275-9da51eb2549a"
Jul 28 01:26:16.980845 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:16.980510    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:16.980845 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:16.980604    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dx5sm" podUID="ddcb15a3-037d-45a0-8787-12585c0d2465"
Jul 28 01:26:16.980845 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:16.980693    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzc9f" podUID="e5ba3944-6409-4af9-9bff-8eb91a65755a"
Jul 28 01:26:18.092034 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:18.091636    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c9ntv" event={"ID":"069b6671-6207-471b-a09c-26a0293ba1ba","Type":"ContainerStarted","Data":"899a0a5fe639c7f7cf4be0ba5cfe35090c998b8d9f9f13783b467751875817cc"}
Jul 28 01:26:18.095451 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:18.095423    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2d5z9" event={"ID":"b8193a25-2add-42b3-8815-022a163695fe","Type":"ContainerStarted","Data":"fa623172ee99c3a6aa9b3bc269472fde3f9112488c0a6516ee532a72bb82ed5d"}
Jul 28 01:26:18.113144 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:18.113092    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2d5z9" podStartSLOduration=7.429620811 podStartE2EDuration="22.113070884s" totalImagesPullingTime="14.683450073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:18.112816025 +0000 UTC m=+21.690273643" watchObservedRunningTime="2026-07-28 01:26:18.113070884 +0000 UTC m=+21.690528502"
Jul 28 01:26:18.113630 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:18.113591    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-81.ec2.internal" podStartSLOduration=20.113578775 podStartE2EDuration="20.113578775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:01.073011541 +0000 UTC m=+4.650469158" watchObservedRunningTime="2026-07-28 01:26:18.113578775 +0000 UTC m=+21.691036393"
Jul 28 01:26:18.980213 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:18.979945    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:18.980412 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:18.979948    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:18.980412 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:18.980325    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpcdj" podUID="cd1a9e35-4116-41f7-b275-9da51eb2549a"
Jul 28 01:26:18.980412 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:18.979947    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:18.980578 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:18.980425    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzc9f" podUID="e5ba3944-6409-4af9-9bff-8eb91a65755a"
Jul 28 01:26:18.980578 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:18.980480    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dx5sm" podUID="ddcb15a3-037d-45a0-8787-12585c0d2465"
Jul 28 01:26:19.098462 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:19.098427    2578 generic.go:356] "Generic (PLEG): container finished" podID="69bc94d5-8275-4e49-b94f-03ef28a1e2b2" containerID="bee13fa4f7c255962cea75550e3e19ad9cb22305aef6997c8c8002a1264bc4ac" exitCode=0
Jul 28 01:26:19.099124 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:19.098501    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwwvx" event={"ID":"69bc94d5-8275-4e49-b94f-03ef28a1e2b2","Type":"ContainerDied","Data":"bee13fa4f7c255962cea75550e3e19ad9cb22305aef6997c8c8002a1264bc4ac"}
Jul 28 01:26:19.100086 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:19.100065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qqq2z" event={"ID":"65e2f293-2ed4-4ff1-aba1-ca582bde2904","Type":"ContainerStarted","Data":"1c9159e7b1a8a0c5683849db544234ef738361387c33126284bd8a8919a0e7f7"}
Jul 28 01:26:19.101674 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:19.101655    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t" event={"ID":"05bfa172-3907-43d0-a248-b4c74dd7620d","Type":"ContainerStarted","Data":"cce3ee50b6c260c2fee1b5346d5c734fe7d8e972935939e9e14497c7dc846bc0"}
Jul 28 01:26:19.102899 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:19.102878    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bh7l6" event={"ID":"5cdf4c08-dce5-43fd-b3b8-c651c49b99c3","Type":"ContainerStarted","Data":"c30615ab43ef295f97bd4ab58af7ef637e88034e8d893f3d38590f0b6612a628"}
Jul 28 01:26:19.104259 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:19.104238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wlzmw" event={"ID":"640c683a-7ed6-4686-a0de-f6c631e1ab3a","Type":"ContainerStarted","Data":"676fcebb1a8d7c0501dde303892d0d9301afe528eff03deab0fa5d8ec190a545"}
Jul 28 01:26:19.107085 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:19.107065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg" event={"ID":"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc","Type":"ContainerStarted","Data":"60aa66e523e5ffcf1b299edeee4c9835a93602670a273e8a134cb5a353973003"}
Jul 28 01:26:19.107170 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:19.107121    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg" event={"ID":"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc","Type":"ContainerStarted","Data":"1d94e34443625a91919f031e814114fc0e85d1fc6f898d75b4f0a3eadb86e761"}
Jul 28 01:26:19.107223 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:19.107194    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg" event={"ID":"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc","Type":"ContainerStarted","Data":"0f2e123087869238721fe461203d088668a0a6263b6fcb3811f0f38e7b35198d"}
Jul 28 01:26:19.107282 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:19.107233    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg" event={"ID":"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc","Type":"ContainerStarted","Data":"552b1379ad115050b2453b11c8774d1baff91d11b8c0c24949e19bc5c2fa724c"}
Jul 28 01:26:19.107282 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:19.107264    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg" event={"ID":"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc","Type":"ContainerStarted","Data":"e45fa37e47f89bd04d80b74b40d1d2c4f1272b8dd4e75fbd9c9e1b69f44ce7a6"}
Jul 28 01:26:19.107402 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:19.107292    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg" event={"ID":"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc","Type":"ContainerStarted","Data":"83b96bcc29160161843493f0fa603c7ac43ed75dfb01b029e7e992d7c88d249d"}
Jul 28 01:26:19.137001 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:19.136956    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c9ntv" podStartSLOduration=3.609084988 podStartE2EDuration="22.136941329s" totalImagesPullingTime="18.527856341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:19.136808316 +0000 UTC m=+22.714265932" watchObservedRunningTime="2026-07-28 01:26:19.136941329 +0000 UTC m=+22.714398946"
Jul 28 01:26:19.154599 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:19.154555    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bh7l6" podStartSLOduration=3.40517397 podStartE2EDuration="22.154540446s" totalImagesPullingTime="18.749366476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:19.154108764 +0000 UTC m=+22.731566398" watchObservedRunningTime="2026-07-28 01:26:19.154540446 +0000 UTC m=+22.731998062"
Jul 28 01:26:19.171216 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:19.171174    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wlzmw" podStartSLOduration=3.407157838 podStartE2EDuration="22.171157718s" totalImagesPullingTime="18.76399988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:19.171078963 +0000 UTC m=+22.748536603" watchObservedRunningTime="2026-07-28 01:26:19.171157718 +0000 UTC m=+22.748615335"
Jul 28 01:26:19.493005 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:19.492743    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 01:26:19.889193 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:19.889085    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T01:26:19.492996823Z","UUID":"1cd44f2b-94aa-4b66-bbc6-12a15a2cbc1c","Handler":null,"Name":"","Endpoint":""}
Jul 28 01:26:19.890822 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:19.890801    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 01:26:19.890948 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:19.890845    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 01:26:20.111971 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:20.111932    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qnv59" event={"ID":"5f007227-cbc6-4d32-a2c8-4ea6e62d606a","Type":"ContainerStarted","Data":"7dcf2d7ebc2f0980a2594f115c34c044163ab22175c0be4feff5b05bc599621d"}
Jul 28 01:26:20.114599 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:20.114569    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t" event={"ID":"05bfa172-3907-43d0-a248-b4c74dd7620d","Type":"ContainerStarted","Data":"5bb7d4820d0a72c29330d809bf716e0bbd793785fcb57097e7b000c919cf28f9"}
Jul 28 01:26:20.126889 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:20.126835    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qqq2z" podStartSLOduration=4.088027218 podStartE2EDuration="23.126817316s" totalImagesPullingTime="19.038790098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:19.19401003 +0000 UTC m=+22.771467658" watchObservedRunningTime="2026-07-28 01:26:20.126817316 +0000 UTC m=+23.704274934"
Jul 28 01:26:20.127323 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:20.127298    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qnv59" podStartSLOduration=4.626614191 podStartE2EDuration="23.12729076s" totalImagesPullingTime="18.500676569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:20.126683781 +0000 UTC m=+23.704141399" watchObservedRunningTime="2026-07-28 01:26:20.12729076 +0000 UTC m=+23.704748378"
Jul 28 01:26:20.980008 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:20.979858    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:20.980008 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:20.979901    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:20.980008 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:20.979858    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:20.980008 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:20.979991    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpcdj" podUID="cd1a9e35-4116-41f7-b275-9da51eb2549a"
Jul 28 01:26:20.980351 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:20.980094    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzc9f" podUID="e5ba3944-6409-4af9-9bff-8eb91a65755a"
Jul 28 01:26:20.980351 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:20.980199    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dx5sm" podUID="ddcb15a3-037d-45a0-8787-12585c0d2465"
Jul 28 01:26:21.120156 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:21.120116    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t" event={"ID":"05bfa172-3907-43d0-a248-b4c74dd7620d","Type":"ContainerStarted","Data":"b40556d35649de7160b69dd249eddedb70f4aee76ea12762971554088f4daa15"}
Jul 28 01:26:21.123928 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:21.123871    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg" event={"ID":"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc","Type":"ContainerStarted","Data":"febc05f797aeb764a682baafeff8123d38af719af52647f2301c543b6f4d88e0"}
Jul 28 01:26:21.138756 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:21.138687    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bmx5t" podStartSLOduration=4.253074278 podStartE2EDuration="25.138670635s" totalImagesPullingTime="20.885596357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:21.138136992 +0000 UTC m=+24.715594608" watchObservedRunningTime="2026-07-28 01:26:21.138670635 +0000 UTC m=+24.716128302"
Jul 28 01:26:22.983280 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:22.983247    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:22.983981 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:22.983392    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:22.983981 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:22.983396    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dx5sm" podUID="ddcb15a3-037d-45a0-8787-12585c0d2465"
Jul 28 01:26:22.983981 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:22.983416    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:22.983981 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:22.983518    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpcdj" podUID="cd1a9e35-4116-41f7-b275-9da51eb2549a"
Jul 28 01:26:22.983981 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:22.983591    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzc9f" podUID="e5ba3944-6409-4af9-9bff-8eb91a65755a"
Jul 28 01:26:23.837186 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:23.836991    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c9ntv"
Jul 28 01:26:23.837712 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:23.837693    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c9ntv"
Jul 28 01:26:24.134096 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:24.134004    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg" event={"ID":"e12a776b-5a0c-40f7-a2d1-221f8dcb27fc","Type":"ContainerStarted","Data":"ff7e5e4665c2661b96da4df50ce0f1c22022f1745bf38051a7b2475fe8080762"}
Jul 28 01:26:24.134782 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:24.134245    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:26:24.136167 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:24.136141    2578 generic.go:356] "Generic (PLEG): container finished" podID="69bc94d5-8275-4e49-b94f-03ef28a1e2b2" containerID="e88674b1df7c4d204907545497c07039db0f616e71a9837d8a52089881b80370" exitCode=0
Jul 28 01:26:24.136291 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:24.136232    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwwvx" event={"ID":"69bc94d5-8275-4e49-b94f-03ef28a1e2b2","Type":"ContainerDied","Data":"e88674b1df7c4d204907545497c07039db0f616e71a9837d8a52089881b80370"}
Jul 28 01:26:24.136359 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:24.136314    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c9ntv"
Jul 28 01:26:24.137558 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:24.136876    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c9ntv"
Jul 28 01:26:24.150697 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:24.150674    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:26:24.173641 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:24.173589    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg" podStartSLOduration=8.451962229 podStartE2EDuration="27.173571893s" totalImagesPullingTime="18.721609664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:24.172181237 +0000 UTC m=+27.749638855" watchObservedRunningTime="2026-07-28 01:26:24.173571893 +0000 UTC m=+27.751029510"
Jul 28 01:26:24.983562 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:24.983492    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:24.983767 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:24.983492    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:24.983767 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:24.983596    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpcdj" podUID="cd1a9e35-4116-41f7-b275-9da51eb2549a"
Jul 28 01:26:24.983767 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:24.983691    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dx5sm" podUID="ddcb15a3-037d-45a0-8787-12585c0d2465"
Jul 28 01:26:24.983767 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:24.983492    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:24.983943 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:24.983782    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzc9f" podUID="e5ba3944-6409-4af9-9bff-8eb91a65755a"
Jul 28 01:26:25.138942 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:25.138913    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:26:25.139464 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:25.138982    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:26:25.154183 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:25.154157    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:26:25.659976 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:25.659747    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zzc9f"]
Jul 28 01:26:25.660174 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:25.660052    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:25.660174 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:25.660145    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzc9f" podUID="e5ba3944-6409-4af9-9bff-8eb91a65755a"
Jul 28 01:26:25.663505 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:25.663475    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dx5sm"]
Jul 28 01:26:25.663639 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:25.663577    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:25.663699 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:25.663670    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dx5sm" podUID="ddcb15a3-037d-45a0-8787-12585c0d2465"
Jul 28 01:26:25.664178 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:25.664154    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wpcdj"]
Jul 28 01:26:25.664283 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:25.664243    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:25.664367 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:25.664343    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpcdj" podUID="cd1a9e35-4116-41f7-b275-9da51eb2549a"
Jul 28 01:26:26.143637 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:26.143606    2578 generic.go:356] "Generic (PLEG): container finished" podID="69bc94d5-8275-4e49-b94f-03ef28a1e2b2" containerID="ed621139ec06e1ecccadf7496945db2c54b123abf31738123a8003595a9c341b" exitCode=0
Jul 28 01:26:26.144144 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:26.143695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwwvx" event={"ID":"69bc94d5-8275-4e49-b94f-03ef28a1e2b2","Type":"ContainerDied","Data":"ed621139ec06e1ecccadf7496945db2c54b123abf31738123a8003595a9c341b"}
Jul 28 01:26:26.981196 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:26.981144    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:26.981320 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:26.981270    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpcdj" podUID="cd1a9e35-4116-41f7-b275-9da51eb2549a"
Jul 28 01:26:26.981412 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:26.981314    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:26.981470 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:26.981448    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzc9f" podUID="e5ba3944-6409-4af9-9bff-8eb91a65755a"
Jul 28 01:26:27.979869 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:27.979838    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:27.980474 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:27.979947    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dx5sm" podUID="ddcb15a3-037d-45a0-8787-12585c0d2465"
Jul 28 01:26:28.150935 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:28.150902    2578 generic.go:356] "Generic (PLEG): container finished" podID="69bc94d5-8275-4e49-b94f-03ef28a1e2b2" containerID="f686e1c5c3a8bf89161763a3afb52a1b643e34da4ff755b505f8216c4fd9dd36" exitCode=0
Jul 28 01:26:28.151098 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:28.150952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwwvx" event={"ID":"69bc94d5-8275-4e49-b94f-03ef28a1e2b2","Type":"ContainerDied","Data":"f686e1c5c3a8bf89161763a3afb52a1b643e34da4ff755b505f8216c4fd9dd36"}
Jul 28 01:26:28.979899 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:28.979862    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:28.980398 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:28.979910    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:28.980398 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:28.980008    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpcdj" podUID="cd1a9e35-4116-41f7-b275-9da51eb2549a"
Jul 28 01:26:28.980398 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:28.980103    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzc9f" podUID="e5ba3944-6409-4af9-9bff-8eb91a65755a"
Jul 28 01:26:29.979734 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:29.979471    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:29.979930 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:29.979832    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dx5sm" podUID="ddcb15a3-037d-45a0-8787-12585c0d2465"
Jul 28 01:26:30.559725 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:30.559684    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs\") pod \"network-metrics-daemon-dx5sm\" (UID: \"ddcb15a3-037d-45a0-8787-12585c0d2465\") " pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:30.560062 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:30.559737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret\") pod \"global-pull-secret-syncer-wpcdj\" (UID: \"cd1a9e35-4116-41f7-b275-9da51eb2549a\") " pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:30.560062 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:30.559854    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:30.560062 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:30.559856    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:30.560062 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:30.559930    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs podName:ddcb15a3-037d-45a0-8787-12585c0d2465 nodeName:}" failed. No retries permitted until 2026-07-28 01:27:02.559909159 +0000 UTC m=+66.137366771 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs") pod "network-metrics-daemon-dx5sm" (UID: "ddcb15a3-037d-45a0-8787-12585c0d2465") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:30.560062 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:30.559948    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret podName:cd1a9e35-4116-41f7-b275-9da51eb2549a nodeName:}" failed. No retries permitted until 2026-07-28 01:27:02.559938948 +0000 UTC m=+66.137396556 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret") pod "global-pull-secret-syncer-wpcdj" (UID: "cd1a9e35-4116-41f7-b275-9da51eb2549a") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:30.660527 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:30.660484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j78fv\" (UniqueName: \"kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv\") pod \"network-check-target-zzc9f\" (UID: \"e5ba3944-6409-4af9-9bff-8eb91a65755a\") " pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:30.660714 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:30.660694    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:30.660766 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:30.660722    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:30.660766 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:30.660737    2578 projected.go:196] Error preparing data for projected volume kube-api-access-j78fv for pod openshift-network-diagnostics/network-check-target-zzc9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:30.660836 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:30.660805    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv podName:e5ba3944-6409-4af9-9bff-8eb91a65755a nodeName:}" failed. No retries permitted until 2026-07-28 01:27:02.660785504 +0000 UTC m=+66.238243114 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-j78fv" (UniqueName: "kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv") pod "network-check-target-zzc9f" (UID: "e5ba3944-6409-4af9-9bff-8eb91a65755a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:30.979996 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:30.979896    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:30.979996 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:30.979931    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:30.980537 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:30.980046    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wpcdj" podUID="cd1a9e35-4116-41f7-b275-9da51eb2549a"
Jul 28 01:26:30.980537 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:30.980183    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzc9f" podUID="e5ba3944-6409-4af9-9bff-8eb91a65755a"
Jul 28 01:26:31.201361 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.201312    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-81.ec2.internal" event="NodeReady"
Jul 28 01:26:31.201558 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.201479    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 01:26:31.260170 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.260090    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-t5gxc"]
Jul 28 01:26:31.262537 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.262483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:26:31.263070 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.263046    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ls4sg"]
Jul 28 01:26:31.265042 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.265020    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ls4sg"
Jul 28 01:26:31.265728 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.265704    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 01:26:31.265844 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.265826    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mh7n2\""
Jul 28 01:26:31.265957 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.265940    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 01:26:31.270091 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.270074    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 01:26:31.270359 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.270345    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 01:26:31.270779 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.270707    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 01:26:31.271002 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.270966    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h497g\""
Jul 28 01:26:31.276109 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.276085    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t5gxc"]
Jul 28 01:26:31.277313 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.277093    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ls4sg"]
Jul 28 01:26:31.365755 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.365712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6862e7b-5460-406a-b6dc-18de72afce4f-tmp-dir\") pod \"dns-default-t5gxc\" (UID: \"b6862e7b-5460-406a-b6dc-18de72afce4f\") " pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:26:31.365929 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.365854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls\") pod \"dns-default-t5gxc\" (UID: \"b6862e7b-5460-406a-b6dc-18de72afce4f\") " pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:26:31.365929 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.365897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6862e7b-5460-406a-b6dc-18de72afce4f-config-volume\") pod \"dns-default-t5gxc\" (UID: \"b6862e7b-5460-406a-b6dc-18de72afce4f\") " pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:26:31.366056 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.365928    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghskd\" (UniqueName: \"kubernetes.io/projected/4678fb71-04c1-4342-aee2-34eb7ac2c129-kube-api-access-ghskd\") pod \"ingress-canary-ls4sg\" (UID: \"4678fb71-04c1-4342-aee2-34eb7ac2c129\") " pod="openshift-ingress-canary/ingress-canary-ls4sg"
Jul 28 01:26:31.366056 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.366014    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8svhl\" (UniqueName: \"kubernetes.io/projected/b6862e7b-5460-406a-b6dc-18de72afce4f-kube-api-access-8svhl\") pod \"dns-default-t5gxc\" (UID: \"b6862e7b-5460-406a-b6dc-18de72afce4f\") " pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:26:31.366147 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.366066    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4678fb71-04c1-4342-aee2-34eb7ac2c129-cert\") pod \"ingress-canary-ls4sg\" (UID: \"4678fb71-04c1-4342-aee2-34eb7ac2c129\") " pod="openshift-ingress-canary/ingress-canary-ls4sg"
Jul 28 01:26:31.466961 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.466924    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8svhl\" (UniqueName: \"kubernetes.io/projected/b6862e7b-5460-406a-b6dc-18de72afce4f-kube-api-access-8svhl\") pod \"dns-default-t5gxc\" (UID: \"b6862e7b-5460-406a-b6dc-18de72afce4f\") " pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:26:31.467147 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.466974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4678fb71-04c1-4342-aee2-34eb7ac2c129-cert\") pod \"ingress-canary-ls4sg\" (UID: \"4678fb71-04c1-4342-aee2-34eb7ac2c129\") " pod="openshift-ingress-canary/ingress-canary-ls4sg"
Jul 28 01:26:31.467147 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.467057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6862e7b-5460-406a-b6dc-18de72afce4f-tmp-dir\") pod \"dns-default-t5gxc\" (UID: \"b6862e7b-5460-406a-b6dc-18de72afce4f\") " pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:26:31.467147 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.467085    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls\") pod \"dns-default-t5gxc\" (UID: \"b6862e7b-5460-406a-b6dc-18de72afce4f\") " pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:26:31.467147 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.467112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6862e7b-5460-406a-b6dc-18de72afce4f-config-volume\") pod \"dns-default-t5gxc\" (UID: \"b6862e7b-5460-406a-b6dc-18de72afce4f\") " pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:26:31.467374 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.467170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghskd\" (UniqueName: \"kubernetes.io/projected/4678fb71-04c1-4342-aee2-34eb7ac2c129-kube-api-access-ghskd\") pod \"ingress-canary-ls4sg\" (UID: \"4678fb71-04c1-4342-aee2-34eb7ac2c129\") " pod="openshift-ingress-canary/ingress-canary-ls4sg"
Jul 28 01:26:31.467374 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:31.467208    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:31.467374 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:31.467270    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls podName:b6862e7b-5460-406a-b6dc-18de72afce4f nodeName:}" failed. No retries permitted until 2026-07-28 01:26:31.967250276 +0000 UTC m=+35.544707874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls") pod "dns-default-t5gxc" (UID: "b6862e7b-5460-406a-b6dc-18de72afce4f") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:31.467636 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.467587    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6862e7b-5460-406a-b6dc-18de72afce4f-tmp-dir\") pod \"dns-default-t5gxc\" (UID: \"b6862e7b-5460-406a-b6dc-18de72afce4f\") " pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:26:31.467822 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.467790    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6862e7b-5460-406a-b6dc-18de72afce4f-config-volume\") pod \"dns-default-t5gxc\" (UID: \"b6862e7b-5460-406a-b6dc-18de72afce4f\") " pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:26:31.472340 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.472305    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4678fb71-04c1-4342-aee2-34eb7ac2c129-cert\") pod \"ingress-canary-ls4sg\" (UID: \"4678fb71-04c1-4342-aee2-34eb7ac2c129\") " pod="openshift-ingress-canary/ingress-canary-ls4sg"
Jul 28 01:26:31.477656 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.477637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8svhl\" (UniqueName: \"kubernetes.io/projected/b6862e7b-5460-406a-b6dc-18de72afce4f-kube-api-access-8svhl\") pod \"dns-default-t5gxc\" (UID: \"b6862e7b-5460-406a-b6dc-18de72afce4f\") " pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:26:31.477994 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.477974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghskd\" (UniqueName: \"kubernetes.io/projected/4678fb71-04c1-4342-aee2-34eb7ac2c129-kube-api-access-ghskd\") pod \"ingress-canary-ls4sg\" (UID: \"4678fb71-04c1-4342-aee2-34eb7ac2c129\") " pod="openshift-ingress-canary/ingress-canary-ls4sg"
Jul 28 01:26:31.582058 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.582020    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ls4sg"
Jul 28 01:26:31.971926 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.971837    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls\") pod \"dns-default-t5gxc\" (UID: \"b6862e7b-5460-406a-b6dc-18de72afce4f\") " pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:26:31.972074 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:31.971996    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:31.972074 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:31.972060    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls podName:b6862e7b-5460-406a-b6dc-18de72afce4f nodeName:}" failed. No retries permitted until 2026-07-28 01:26:32.97204314 +0000 UTC m=+36.549500743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls") pod "dns-default-t5gxc" (UID: "b6862e7b-5460-406a-b6dc-18de72afce4f") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:31.979377 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.979349    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:26:31.982698 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.982437    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5299b\""
Jul 28 01:26:31.982698 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:31.982553    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 01:26:32.482388 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:32.482208    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ls4sg"]
Jul 28 01:26:32.485571 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:26:32.485541    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4678fb71_04c1_4342_aee2_34eb7ac2c129.slice/crio-00b640a78cd6c773dd3d286f37fbcc78145443e3e811cccba17ba26fe19d8664 WatchSource:0}: Error finding container 00b640a78cd6c773dd3d286f37fbcc78145443e3e811cccba17ba26fe19d8664: Status 404 returned error can't find the container with id 00b640a78cd6c773dd3d286f37fbcc78145443e3e811cccba17ba26fe19d8664
Jul 28 01:26:32.980202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:32.980160    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:26:32.980425 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:32.980209    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:26:32.980425 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:32.980226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls\") pod \"dns-default-t5gxc\" (UID: \"b6862e7b-5460-406a-b6dc-18de72afce4f\") " pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:26:32.980425 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:32.980354    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:32.980425 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:32.980417    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls podName:b6862e7b-5460-406a-b6dc-18de72afce4f nodeName:}" failed. No retries permitted until 2026-07-28 01:26:34.980397926 +0000 UTC m=+38.557855525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls") pod "dns-default-t5gxc" (UID: "b6862e7b-5460-406a-b6dc-18de72afce4f") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:32.986983 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:32.986755    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 01:26:32.986983 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:32.986790    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 01:26:32.988110 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:32.987896    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4mq45\""
Jul 28 01:26:32.988110 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:32.987932    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 01:26:33.164662 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:33.164627    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ls4sg" event={"ID":"4678fb71-04c1-4342-aee2-34eb7ac2c129","Type":"ContainerStarted","Data":"00b640a78cd6c773dd3d286f37fbcc78145443e3e811cccba17ba26fe19d8664"}
Jul 28 01:26:33.167710 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:33.167680    2578 generic.go:356] "Generic (PLEG): container finished" podID="69bc94d5-8275-4e49-b94f-03ef28a1e2b2" containerID="c2d346b56224064e09ed641d1ca8fa0a0d25be2346300ef868c51ce889ed8bb0" exitCode=0
Jul 28 01:26:33.167855 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:33.167731    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwwvx" event={"ID":"69bc94d5-8275-4e49-b94f-03ef28a1e2b2","Type":"ContainerDied","Data":"c2d346b56224064e09ed641d1ca8fa0a0d25be2346300ef868c51ce889ed8bb0"}
Jul 28 01:26:34.174154 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:34.174115    2578 generic.go:356] "Generic (PLEG): container finished" podID="69bc94d5-8275-4e49-b94f-03ef28a1e2b2" containerID="9d99244f3b6d528e8159a344613cc260e9ddf3b98553e2aeb9fc21537a553ed0" exitCode=0
Jul 28 01:26:34.174793 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:34.174162    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwwvx" event={"ID":"69bc94d5-8275-4e49-b94f-03ef28a1e2b2","Type":"ContainerDied","Data":"9d99244f3b6d528e8159a344613cc260e9ddf3b98553e2aeb9fc21537a553ed0"}
Jul 28 01:26:34.997499 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:34.997464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls\") pod \"dns-default-t5gxc\" (UID: \"b6862e7b-5460-406a-b6dc-18de72afce4f\") " pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:26:34.997650 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:34.997604    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:34.997716 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:34.997666    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls podName:b6862e7b-5460-406a-b6dc-18de72afce4f nodeName:}" failed. No retries permitted until 2026-07-28 01:26:38.997647733 +0000 UTC m=+42.575105347 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls") pod "dns-default-t5gxc" (UID: "b6862e7b-5460-406a-b6dc-18de72afce4f") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:35.177922 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:35.177828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ls4sg" event={"ID":"4678fb71-04c1-4342-aee2-34eb7ac2c129","Type":"ContainerStarted","Data":"af35ffb82675b40edd5457f586f23e3955ed877bf84c3f1cc033ca1ccdb04014"}
Jul 28 01:26:35.181270 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:35.181243    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gwwvx" event={"ID":"69bc94d5-8275-4e49-b94f-03ef28a1e2b2","Type":"ContainerStarted","Data":"977a615b71d43a6339e655cfd63c6f6cd56df8fb1af46605595709537daa2ed7"}
Jul 28 01:26:35.197510 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:35.197458    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ls4sg" podStartSLOduration=1.782890938 podStartE2EDuration="4.197442717s" totalImagesPullingTime="2.414551779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:35.196931816 +0000 UTC m=+38.774389433" watchObservedRunningTime="2026-07-28 01:26:35.197442717 +0000 UTC m=+38.774900311"
Jul 28 01:26:35.220798 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:35.220751    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gwwvx" podStartSLOduration=9.182638323 podStartE2EDuration="38.220734085s" totalImagesPullingTime="29.038095762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:35.218645255 +0000 UTC m=+38.796102898" watchObservedRunningTime="2026-07-28 01:26:35.220734085 +0000 UTC m=+38.798191710"
Jul 28 01:26:37.824241 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.824206    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2"]
Jul 28 01:26:37.866135 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.866097    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-68787665fc-c2vr7"]
Jul 28 01:26:37.866275 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.866225    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2"
Jul 28 01:26:37.868669 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.868647    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 28 01:26:37.869205 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.869183    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-bbhcv\""
Jul 28 01:26:37.869343 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.869221    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 28 01:26:37.869343 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.869263    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:26:37.869462 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.869435    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 28 01:26:37.889804 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.889779    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68787665fc-c2vr7"]
Jul 28 01:26:37.889804 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.889809    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2"]
Jul 28 01:26:37.889961 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.889884    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:37.893478 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.893454    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 28 01:26:37.893618 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.893495    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 28 01:26:37.893618 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.893495    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 28 01:26:37.893618 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.893532    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 01:26:37.893774 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.893619    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 28 01:26:37.893829 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.893785    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-pczwk\""
Jul 28 01:26:37.893915 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.893899    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 01:26:37.924468 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.924440    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p"]
Jul 28 01:26:37.945681 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.945646    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p"]
Jul 28 01:26:37.945847 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.945755    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p"
Jul 28 01:26:37.948779 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.948754    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:26:37.948914 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.948875    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-29g9h\""
Jul 28 01:26:37.948914 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.948901    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 28 01:26:37.949018 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:37.948979    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 28 01:26:38.016159 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.016118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79ec8bac-483f-4773-a8b0-5ac56d23c8bb-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2\" (UID: \"79ec8bac-483f-4773-a8b0-5ac56d23c8bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2"
Jul 28 01:26:38.016159 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.016155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-default-certificate\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:38.016395 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.016181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkt6r\" (UniqueName: \"kubernetes.io/projected/79ec8bac-483f-4773-a8b0-5ac56d23c8bb-kube-api-access-dkt6r\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2\" (UID: \"79ec8bac-483f-4773-a8b0-5ac56d23c8bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2"
Jul 28 01:26:38.016395 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.016252    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5p69\" (UniqueName: \"kubernetes.io/projected/e3932682-22fe-44a7-93ae-6de33ab1ed72-kube-api-access-w5p69\") pod \"cluster-samples-operator-6748489656-2gt7p\" (UID: \"e3932682-22fe-44a7-93ae-6de33ab1ed72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p"
Jul 28 01:26:38.016395 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.016291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:38.016497 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.016391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwlfj\" (UniqueName: \"kubernetes.io/projected/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-kube-api-access-nwlfj\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:38.016497 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.016417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3932682-22fe-44a7-93ae-6de33ab1ed72-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2gt7p\" (UID: \"e3932682-22fe-44a7-93ae-6de33ab1ed72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p"
Jul 28 01:26:38.016497 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.016446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/79ec8bac-483f-4773-a8b0-5ac56d23c8bb-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2\" (UID: \"79ec8bac-483f-4773-a8b0-5ac56d23c8bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2"
Jul 28 01:26:38.016497 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.016466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-metrics-certs\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:38.016497 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.016489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-stats-auth\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:38.117282 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.117194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkt6r\" (UniqueName: \"kubernetes.io/projected/79ec8bac-483f-4773-a8b0-5ac56d23c8bb-kube-api-access-dkt6r\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2\" (UID: \"79ec8bac-483f-4773-a8b0-5ac56d23c8bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2"
Jul 28 01:26:38.117282 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.117234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5p69\" (UniqueName: \"kubernetes.io/projected/e3932682-22fe-44a7-93ae-6de33ab1ed72-kube-api-access-w5p69\") pod \"cluster-samples-operator-6748489656-2gt7p\" (UID: \"e3932682-22fe-44a7-93ae-6de33ab1ed72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p"
Jul 28 01:26:38.117282 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.117259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:38.117582 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.117320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwlfj\" (UniqueName: \"kubernetes.io/projected/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-kube-api-access-nwlfj\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:38.117582 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.117384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3932682-22fe-44a7-93ae-6de33ab1ed72-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2gt7p\" (UID: \"e3932682-22fe-44a7-93ae-6de33ab1ed72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p"
Jul 28 01:26:38.117582 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.117423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/79ec8bac-483f-4773-a8b0-5ac56d23c8bb-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2\" (UID: \"79ec8bac-483f-4773-a8b0-5ac56d23c8bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2"
Jul 28 01:26:38.117582 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.117452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-metrics-certs\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:38.117582 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:38.117477    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle podName:3f8169c2-427b-49b7-a2cc-8e8844f0d80c nodeName:}" failed. No retries permitted until 2026-07-28 01:26:38.617454565 +0000 UTC m=+42.194912180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle") pod "router-default-68787665fc-c2vr7" (UID: "3f8169c2-427b-49b7-a2cc-8e8844f0d80c") : configmap references non-existent config key: service-ca.crt
Jul 28 01:26:38.117582 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:38.117516    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 01:26:38.117582 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:38.117526    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 01:26:38.117582 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.117527    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-stats-auth\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:38.117582 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:38.117585    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3932682-22fe-44a7-93ae-6de33ab1ed72-samples-operator-tls podName:e3932682-22fe-44a7-93ae-6de33ab1ed72 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:38.617566853 +0000 UTC m=+42.195024464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3932682-22fe-44a7-93ae-6de33ab1ed72-samples-operator-tls") pod "cluster-samples-operator-6748489656-2gt7p" (UID: "e3932682-22fe-44a7-93ae-6de33ab1ed72") : secret "samples-operator-tls" not found
Jul 28 01:26:38.118060 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:38.117619    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-metrics-certs podName:3f8169c2-427b-49b7-a2cc-8e8844f0d80c nodeName:}" failed. No retries permitted until 2026-07-28 01:26:38.617600393 +0000 UTC m=+42.195057993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-metrics-certs") pod "router-default-68787665fc-c2vr7" (UID: "3f8169c2-427b-49b7-a2cc-8e8844f0d80c") : secret "router-metrics-certs-default" not found
Jul 28 01:26:38.118060 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.117689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79ec8bac-483f-4773-a8b0-5ac56d23c8bb-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2\" (UID: \"79ec8bac-483f-4773-a8b0-5ac56d23c8bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2"
Jul 28 01:26:38.118060 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.117720    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-default-certificate\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:38.118060 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.118052    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/79ec8bac-483f-4773-a8b0-5ac56d23c8bb-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2\" (UID: \"79ec8bac-483f-4773-a8b0-5ac56d23c8bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2"
Jul 28 01:26:38.120819 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.120793    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79ec8bac-483f-4773-a8b0-5ac56d23c8bb-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2\" (UID: \"79ec8bac-483f-4773-a8b0-5ac56d23c8bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2"
Jul 28 01:26:38.120992 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.120975    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-default-certificate\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:38.121031 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.120977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-stats-auth\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:38.128639 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.128616    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwlfj\" (UniqueName: \"kubernetes.io/projected/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-kube-api-access-nwlfj\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:38.128919 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.128898    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkt6r\" (UniqueName: \"kubernetes.io/projected/79ec8bac-483f-4773-a8b0-5ac56d23c8bb-kube-api-access-dkt6r\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2\" (UID: \"79ec8bac-483f-4773-a8b0-5ac56d23c8bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2"
Jul 28 01:26:38.128960 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.128940    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5p69\" (UniqueName: \"kubernetes.io/projected/e3932682-22fe-44a7-93ae-6de33ab1ed72-kube-api-access-w5p69\") pod \"cluster-samples-operator-6748489656-2gt7p\" (UID: \"e3932682-22fe-44a7-93ae-6de33ab1ed72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p"
Jul 28 01:26:38.176141 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.176105    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2"
Jul 28 01:26:38.325975 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.325941    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2"]
Jul 28 01:26:38.328787 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:26:38.328761    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ec8bac_483f_4773_a8b0_5ac56d23c8bb.slice/crio-4c6ee51edf018009f35851718e8cc6ebd34142bc4dc2e4b1966ae1e522ca4570 WatchSource:0}: Error finding container 4c6ee51edf018009f35851718e8cc6ebd34142bc4dc2e4b1966ae1e522ca4570: Status 404 returned error can't find the container with id 4c6ee51edf018009f35851718e8cc6ebd34142bc4dc2e4b1966ae1e522ca4570
Jul 28 01:26:38.622391 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.622356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:38.622581 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.622411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3932682-22fe-44a7-93ae-6de33ab1ed72-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2gt7p\" (UID: \"e3932682-22fe-44a7-93ae-6de33ab1ed72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p"
Jul 28 01:26:38.622581 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:38.622437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-metrics-certs\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:38.622581 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:38.622538    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 01:26:38.622581 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:38.622540    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle podName:3f8169c2-427b-49b7-a2cc-8e8844f0d80c nodeName:}" failed. No retries permitted until 2026-07-28 01:26:39.622518209 +0000 UTC m=+43.199975825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle") pod "router-default-68787665fc-c2vr7" (UID: "3f8169c2-427b-49b7-a2cc-8e8844f0d80c") : configmap references non-existent config key: service-ca.crt
Jul 28 01:26:38.622581 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:38.622553    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 01:26:38.622739 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:38.622591    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-metrics-certs podName:3f8169c2-427b-49b7-a2cc-8e8844f0d80c nodeName:}" failed. No retries permitted until 2026-07-28 01:26:39.622576379 +0000 UTC m=+43.200033992 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-metrics-certs") pod "router-default-68787665fc-c2vr7" (UID: "3f8169c2-427b-49b7-a2cc-8e8844f0d80c") : secret "router-metrics-certs-default" not found
Jul 28 01:26:38.622739 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:38.622608    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3932682-22fe-44a7-93ae-6de33ab1ed72-samples-operator-tls podName:e3932682-22fe-44a7-93ae-6de33ab1ed72 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:39.622599477 +0000 UTC m=+43.200057075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3932682-22fe-44a7-93ae-6de33ab1ed72-samples-operator-tls") pod "cluster-samples-operator-6748489656-2gt7p" (UID: "e3932682-22fe-44a7-93ae-6de33ab1ed72") : secret "samples-operator-tls" not found
Jul 28 01:26:39.025356 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:39.025314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls\") pod \"dns-default-t5gxc\" (UID: \"b6862e7b-5460-406a-b6dc-18de72afce4f\") " pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:26:39.025812 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:39.025507    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:39.025812 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:39.025594    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls podName:b6862e7b-5460-406a-b6dc-18de72afce4f nodeName:}" failed. No retries permitted until 2026-07-28 01:26:47.025571847 +0000 UTC m=+50.603029455 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls") pod "dns-default-t5gxc" (UID: "b6862e7b-5460-406a-b6dc-18de72afce4f") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:39.192101 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:39.192056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2" event={"ID":"79ec8bac-483f-4773-a8b0-5ac56d23c8bb","Type":"ContainerStarted","Data":"4c6ee51edf018009f35851718e8cc6ebd34142bc4dc2e4b1966ae1e522ca4570"}
Jul 28 01:26:39.630937 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:39.630901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:39.631118 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:39.630983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3932682-22fe-44a7-93ae-6de33ab1ed72-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2gt7p\" (UID: \"e3932682-22fe-44a7-93ae-6de33ab1ed72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p"
Jul 28 01:26:39.631118 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:39.631008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-metrics-certs\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:39.631118 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:39.631088    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle podName:3f8169c2-427b-49b7-a2cc-8e8844f0d80c nodeName:}" failed. No retries permitted until 2026-07-28 01:26:41.631070016 +0000 UTC m=+45.208527611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle") pod "router-default-68787665fc-c2vr7" (UID: "3f8169c2-427b-49b7-a2cc-8e8844f0d80c") : configmap references non-existent config key: service-ca.crt
Jul 28 01:26:39.631277 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:39.631132    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 01:26:39.631277 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:39.631162    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 01:26:39.631277 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:39.631196    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-metrics-certs podName:3f8169c2-427b-49b7-a2cc-8e8844f0d80c nodeName:}" failed. No retries permitted until 2026-07-28 01:26:41.631179165 +0000 UTC m=+45.208636763 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-metrics-certs") pod "router-default-68787665fc-c2vr7" (UID: "3f8169c2-427b-49b7-a2cc-8e8844f0d80c") : secret "router-metrics-certs-default" not found
Jul 28 01:26:39.631277 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:39.631215    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3932682-22fe-44a7-93ae-6de33ab1ed72-samples-operator-tls podName:e3932682-22fe-44a7-93ae-6de33ab1ed72 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:41.63120522 +0000 UTC m=+45.208662814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3932682-22fe-44a7-93ae-6de33ab1ed72-samples-operator-tls") pod "cluster-samples-operator-6748489656-2gt7p" (UID: "e3932682-22fe-44a7-93ae-6de33ab1ed72") : secret "samples-operator-tls" not found
Jul 28 01:26:41.198638 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:41.198596    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2" event={"ID":"79ec8bac-483f-4773-a8b0-5ac56d23c8bb","Type":"ContainerStarted","Data":"66332cdfe0a8f966c0fd57692eccf8bac07309b26cf9b834b533c8cdde257aa6"}
Jul 28 01:26:41.217752 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:41.217705    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2" podStartSLOduration=1.9729801139999998 podStartE2EDuration="4.217689447s" totalImagesPullingTime="2.244709333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:41.217258945 +0000 UTC m=+44.794716575" watchObservedRunningTime="2026-07-28 01:26:41.217689447 +0000 UTC m=+44.795147063"
Jul 28 01:26:41.648393 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:41.648354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3932682-22fe-44a7-93ae-6de33ab1ed72-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2gt7p\" (UID: \"e3932682-22fe-44a7-93ae-6de33ab1ed72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p"
Jul 28 01:26:41.648393 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:41.648401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-metrics-certs\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:41.648673 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:41.648461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:41.648673 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:41.648527    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 01:26:41.648673 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:41.648564    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 01:26:41.648673 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:41.648612    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle podName:3f8169c2-427b-49b7-a2cc-8e8844f0d80c nodeName:}" failed. No retries permitted until 2026-07-28 01:26:45.648593999 +0000 UTC m=+49.226051614 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle") pod "router-default-68787665fc-c2vr7" (UID: "3f8169c2-427b-49b7-a2cc-8e8844f0d80c") : configmap references non-existent config key: service-ca.crt
Jul 28 01:26:41.648673 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:41.648628    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-metrics-certs podName:3f8169c2-427b-49b7-a2cc-8e8844f0d80c nodeName:}" failed. No retries permitted until 2026-07-28 01:26:45.648621431 +0000 UTC m=+49.226079025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-metrics-certs") pod "router-default-68787665fc-c2vr7" (UID: "3f8169c2-427b-49b7-a2cc-8e8844f0d80c") : secret "router-metrics-certs-default" not found
Jul 28 01:26:41.648673 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:41.648639    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3932682-22fe-44a7-93ae-6de33ab1ed72-samples-operator-tls podName:e3932682-22fe-44a7-93ae-6de33ab1ed72 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:45.648633976 +0000 UTC m=+49.226091571 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3932682-22fe-44a7-93ae-6de33ab1ed72-samples-operator-tls") pod "cluster-samples-operator-6748489656-2gt7p" (UID: "e3932682-22fe-44a7-93ae-6de33ab1ed72") : secret "samples-operator-tls" not found
Jul 28 01:26:41.974253 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:41.974172    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-rkmdz"]
Jul 28 01:26:41.994759 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:41.994727    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-rkmdz"]
Jul 28 01:26:41.994906 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:41.994854    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-rkmdz"
Jul 28 01:26:41.998037 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:41.998014    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 01:26:41.998037 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:41.998014    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-w9xb7\""
Jul 28 01:26:41.999384 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:41.999362    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 01:26:42.152534 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:42.152499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7844\" (UniqueName: \"kubernetes.io/projected/b4d9fecf-bea9-4cd4-977a-ae015ee24e38-kube-api-access-r7844\") pod \"migrator-6865c9967-rkmdz\" (UID: \"b4d9fecf-bea9-4cd4-977a-ae015ee24e38\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-rkmdz"
Jul 28 01:26:42.253541 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:42.253433    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7844\" (UniqueName: \"kubernetes.io/projected/b4d9fecf-bea9-4cd4-977a-ae015ee24e38-kube-api-access-r7844\") pod \"migrator-6865c9967-rkmdz\" (UID: \"b4d9fecf-bea9-4cd4-977a-ae015ee24e38\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-rkmdz"
Jul 28 01:26:42.263630 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:42.263594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7844\" (UniqueName: \"kubernetes.io/projected/b4d9fecf-bea9-4cd4-977a-ae015ee24e38-kube-api-access-r7844\") pod \"migrator-6865c9967-rkmdz\" (UID: \"b4d9fecf-bea9-4cd4-977a-ae015ee24e38\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-rkmdz"
Jul 28 01:26:42.304014 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:42.303983    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-rkmdz"
Jul 28 01:26:42.450348 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:42.450301    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-rkmdz"]
Jul 28 01:26:42.453193 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:26:42.453170    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d9fecf_bea9_4cd4_977a_ae015ee24e38.slice/crio-1d610c1a6d486d7e476ef6c80549cddb8c31f34c6af4b796d64eedc5054c09c2 WatchSource:0}: Error finding container 1d610c1a6d486d7e476ef6c80549cddb8c31f34c6af4b796d64eedc5054c09c2: Status 404 returned error can't find the container with id 1d610c1a6d486d7e476ef6c80549cddb8c31f34c6af4b796d64eedc5054c09c2
Jul 28 01:26:43.204482 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:43.204447    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-rkmdz" event={"ID":"b4d9fecf-bea9-4cd4-977a-ae015ee24e38","Type":"ContainerStarted","Data":"1d610c1a6d486d7e476ef6c80549cddb8c31f34c6af4b796d64eedc5054c09c2"}
Jul 28 01:26:44.208295 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:44.208263    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-rkmdz" event={"ID":"b4d9fecf-bea9-4cd4-977a-ae015ee24e38","Type":"ContainerStarted","Data":"5bf06e2f452b2977d49c4bd8bb09ae16ed88a000e0a959d399107fc3367b402a"}
Jul 28 01:26:45.212126 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.212085    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-rkmdz" event={"ID":"b4d9fecf-bea9-4cd4-977a-ae015ee24e38","Type":"ContainerStarted","Data":"f650b83fb3a55de57eaa748c0c036b065c0c6bb2393502a1395bd2057e6f2e6f"}
Jul 28 01:26:45.230774 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.230714    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-rkmdz" podStartSLOduration=2.7164598509999998 podStartE2EDuration="4.230692867s" totalImagesPullingTime="1.514233016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:45.22872906 +0000 UTC m=+48.806186678" watchObservedRunningTime="2026-07-28 01:26:45.230692867 +0000 UTC m=+48.808150486"
Jul 28 01:26:45.509239 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.509167    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mxrqp"]
Jul 28 01:26:45.532918 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.532894    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mxrqp"]
Jul 28 01:26:45.533062 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.532989    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:45.535804 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.535782    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 01:26:45.535920 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.535892    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gqjq9\""
Jul 28 01:26:45.537134 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.537115    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 01:26:45.537134 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.537131    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 01:26:45.537254 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.537194    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 01:26:45.679766 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.679730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:45.679944 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.679772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5ac739c6-6fe9-43a1-a92d-e1298aea7774-crio-socket\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:45.679944 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.679802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ac739c6-6fe9-43a1-a92d-e1298aea7774-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:45.679944 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.679873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ac739c6-6fe9-43a1-a92d-e1298aea7774-data-volume\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:45.679944 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.679917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:45.680121 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.679962    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3932682-22fe-44a7-93ae-6de33ab1ed72-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2gt7p\" (UID: \"e3932682-22fe-44a7-93ae-6de33ab1ed72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p"
Jul 28 01:26:45.680121 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.679989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-metrics-certs\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:45.680121 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.680008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjvk4\" (UniqueName: \"kubernetes.io/projected/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-api-access-mjvk4\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:45.680121 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:45.680097    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 01:26:45.680121 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:45.680118    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 01:26:45.680325 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:45.680121    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle podName:3f8169c2-427b-49b7-a2cc-8e8844f0d80c nodeName:}" failed. No retries permitted until 2026-07-28 01:26:53.680099399 +0000 UTC m=+57.257556994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle") pod "router-default-68787665fc-c2vr7" (UID: "3f8169c2-427b-49b7-a2cc-8e8844f0d80c") : configmap references non-existent config key: service-ca.crt
Jul 28 01:26:45.680325 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:45.680194    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3932682-22fe-44a7-93ae-6de33ab1ed72-samples-operator-tls podName:e3932682-22fe-44a7-93ae-6de33ab1ed72 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:53.680176691 +0000 UTC m=+57.257634286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e3932682-22fe-44a7-93ae-6de33ab1ed72-samples-operator-tls") pod "cluster-samples-operator-6748489656-2gt7p" (UID: "e3932682-22fe-44a7-93ae-6de33ab1ed72") : secret "samples-operator-tls" not found
Jul 28 01:26:45.680325 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:45.680218    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-metrics-certs podName:3f8169c2-427b-49b7-a2cc-8e8844f0d80c nodeName:}" failed. No retries permitted until 2026-07-28 01:26:53.680208742 +0000 UTC m=+57.257666342 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-metrics-certs") pod "router-default-68787665fc-c2vr7" (UID: "3f8169c2-427b-49b7-a2cc-8e8844f0d80c") : secret "router-metrics-certs-default" not found
Jul 28 01:26:45.781067 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.781033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvk4\" (UniqueName: \"kubernetes.io/projected/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-api-access-mjvk4\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:45.781274 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.781092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:45.781274 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:45.781152    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:45.781274 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:45.781205    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-rbac-proxy-cm podName:5ac739c6-6fe9-43a1-a92d-e1298aea7774 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:46.281186629 +0000 UTC m=+49.858644226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mxrqp" (UID: "5ac739c6-6fe9-43a1-a92d-e1298aea7774") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:45.781274 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.781218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5ac739c6-6fe9-43a1-a92d-e1298aea7774-crio-socket\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:45.781274 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.781234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ac739c6-6fe9-43a1-a92d-e1298aea7774-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:45.781274 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.781269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ac739c6-6fe9-43a1-a92d-e1298aea7774-data-volume\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:45.781578 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:45.781384    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:45.781578 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:45.781435    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ac739c6-6fe9-43a1-a92d-e1298aea7774-insights-runtime-extractor-tls podName:5ac739c6-6fe9-43a1-a92d-e1298aea7774 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:46.281423487 +0000 UTC m=+49.858881085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5ac739c6-6fe9-43a1-a92d-e1298aea7774-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mxrqp" (UID: "5ac739c6-6fe9-43a1-a92d-e1298aea7774") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:45.781578 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.781467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5ac739c6-6fe9-43a1-a92d-e1298aea7774-crio-socket\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:45.781685 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.781600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ac739c6-6fe9-43a1-a92d-e1298aea7774-data-volume\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:45.801021 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:45.800987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjvk4\" (UniqueName: \"kubernetes.io/projected/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-api-access-mjvk4\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:46.286471 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:46.286433    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:46.286471 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:46.286475    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ac739c6-6fe9-43a1-a92d-e1298aea7774-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:46.286917 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:46.286565    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:46.286917 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:46.286634    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:46.286917 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:46.286668    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-rbac-proxy-cm podName:5ac739c6-6fe9-43a1-a92d-e1298aea7774 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:47.28664968 +0000 UTC m=+50.864107275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mxrqp" (UID: "5ac739c6-6fe9-43a1-a92d-e1298aea7774") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:46.286917 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:46.286682    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ac739c6-6fe9-43a1-a92d-e1298aea7774-insights-runtime-extractor-tls podName:5ac739c6-6fe9-43a1-a92d-e1298aea7774 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:47.286676232 +0000 UTC m=+50.864133827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5ac739c6-6fe9-43a1-a92d-e1298aea7774-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mxrqp" (UID: "5ac739c6-6fe9-43a1-a92d-e1298aea7774") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:47.092539 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:47.092505    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls\") pod \"dns-default-t5gxc\" (UID: \"b6862e7b-5460-406a-b6dc-18de72afce4f\") " pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:26:47.092733 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:47.092648    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 01:26:47.092733 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:47.092716    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls podName:b6862e7b-5460-406a-b6dc-18de72afce4f nodeName:}" failed. No retries permitted until 2026-07-28 01:27:03.092699412 +0000 UTC m=+66.670157008 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls") pod "dns-default-t5gxc" (UID: "b6862e7b-5460-406a-b6dc-18de72afce4f") : secret "dns-default-metrics-tls" not found
Jul 28 01:26:47.294737 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:47.294702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:47.295081 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:47.294745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ac739c6-6fe9-43a1-a92d-e1298aea7774-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:47.295081 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:47.294823    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:47.295081 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:47.294851    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:47.295081 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:47.294896    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-rbac-proxy-cm podName:5ac739c6-6fe9-43a1-a92d-e1298aea7774 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:49.294872908 +0000 UTC m=+52.872330526 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mxrqp" (UID: "5ac739c6-6fe9-43a1-a92d-e1298aea7774") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:47.295081 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:47.294912    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ac739c6-6fe9-43a1-a92d-e1298aea7774-insights-runtime-extractor-tls podName:5ac739c6-6fe9-43a1-a92d-e1298aea7774 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:49.294904328 +0000 UTC m=+52.872361922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5ac739c6-6fe9-43a1-a92d-e1298aea7774-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mxrqp" (UID: "5ac739c6-6fe9-43a1-a92d-e1298aea7774") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:47.683144 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:47.683119    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wlzmw_640c683a-7ed6-4686-a0de-f6c631e1ab3a/dns-node-resolver/0.log"
Jul 28 01:26:48.081135 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:48.081104    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2d5z9_b8193a25-2add-42b3-8815-022a163695fe/node-ca/0.log"
Jul 28 01:26:49.080525 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:49.080496    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ls4sg_4678fb71-04c1-4342-aee2-34eb7ac2c129/serve-healthcheck-canary/0.log"
Jul 28 01:26:49.310889 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:49.310849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:49.310889 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:49.310893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ac739c6-6fe9-43a1-a92d-e1298aea7774-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:49.311174 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:49.310981    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:49.311174 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:49.311045    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-rbac-proxy-cm podName:5ac739c6-6fe9-43a1-a92d-e1298aea7774 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:53.311030642 +0000 UTC m=+56.888488237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mxrqp" (UID: "5ac739c6-6fe9-43a1-a92d-e1298aea7774") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:49.311174 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:49.310985    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:49.311174 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:49.311108    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ac739c6-6fe9-43a1-a92d-e1298aea7774-insights-runtime-extractor-tls podName:5ac739c6-6fe9-43a1-a92d-e1298aea7774 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:53.31109728 +0000 UTC m=+56.888554875 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5ac739c6-6fe9-43a1-a92d-e1298aea7774-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mxrqp" (UID: "5ac739c6-6fe9-43a1-a92d-e1298aea7774") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:53.341646 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:53.341591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:53.342262 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:53.341662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ac739c6-6fe9-43a1-a92d-e1298aea7774-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:26:53.342262 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:53.341719    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 01:26:53.342262 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:53.341793    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-rbac-proxy-cm podName:5ac739c6-6fe9-43a1-a92d-e1298aea7774 nodeName:}" failed. No retries permitted until 2026-07-28 01:27:01.341769415 +0000 UTC m=+64.919227009 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mxrqp" (UID: "5ac739c6-6fe9-43a1-a92d-e1298aea7774") : configmap "kube-rbac-proxy" not found
Jul 28 01:26:53.342262 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:53.341803    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:53.342262 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:53.341862    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ac739c6-6fe9-43a1-a92d-e1298aea7774-insights-runtime-extractor-tls podName:5ac739c6-6fe9-43a1-a92d-e1298aea7774 nodeName:}" failed. No retries permitted until 2026-07-28 01:27:01.34184613 +0000 UTC m=+64.919303726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5ac739c6-6fe9-43a1-a92d-e1298aea7774-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mxrqp" (UID: "5ac739c6-6fe9-43a1-a92d-e1298aea7774") : secret "insights-runtime-extractor-tls" not found
Jul 28 01:26:53.744404 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:53.744297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:53.744559 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:53.744442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3932682-22fe-44a7-93ae-6de33ab1ed72-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2gt7p\" (UID: \"e3932682-22fe-44a7-93ae-6de33ab1ed72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p"
Jul 28 01:26:53.744559 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:26:53.744467    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle podName:3f8169c2-427b-49b7-a2cc-8e8844f0d80c nodeName:}" failed. No retries permitted until 2026-07-28 01:27:09.744446308 +0000 UTC m=+73.321903904 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle") pod "router-default-68787665fc-c2vr7" (UID: "3f8169c2-427b-49b7-a2cc-8e8844f0d80c") : configmap references non-existent config key: service-ca.crt
Jul 28 01:26:53.744559 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:53.744521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-metrics-certs\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:53.746855 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:53.746824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3932682-22fe-44a7-93ae-6de33ab1ed72-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2gt7p\" (UID: \"e3932682-22fe-44a7-93ae-6de33ab1ed72\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p"
Jul 28 01:26:53.746968 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:53.746824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-metrics-certs\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:26:53.854550 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:53.854513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p"
Jul 28 01:26:54.002515 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:54.002430    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p"]
Jul 28 01:26:54.233045 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:54.233010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p" event={"ID":"e3932682-22fe-44a7-93ae-6de33ab1ed72","Type":"ContainerStarted","Data":"0f2f6fc24eaf7614eb08d55378c99db7ad4f1d0f3df7a644382dcd13ca4026c4"}
Jul 28 01:26:56.240484 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:56.240443    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p" event={"ID":"e3932682-22fe-44a7-93ae-6de33ab1ed72","Type":"ContainerStarted","Data":"87686f94645a5dae57a33a731297bd1bba4115bfa9e7fb7020522b9332b06292"}
Jul 28 01:26:56.240867 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:56.240511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p" event={"ID":"e3932682-22fe-44a7-93ae-6de33ab1ed72","Type":"ContainerStarted","Data":"71153fd43f703cbed6e72b4499ada709c564d4bee45216fe466bff28b717b33d"}
Jul 28 01:26:56.275311 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:56.275261    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2gt7p" podStartSLOduration=17.84597047 podStartE2EDuration="19.275244572s" totalImagesPullingTime="1.429274102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:56.274549558 +0000 UTC m=+59.852007175" watchObservedRunningTime="2026-07-28 01:26:56.275244572 +0000 UTC m=+59.852702188"
Jul 28 01:26:57.164849 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:26:57.164820    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5mxg"
Jul 28 01:27:01.406682 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:01.406641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:27:01.406682 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:01.406687    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ac739c6-6fe9-43a1-a92d-e1298aea7774-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:27:01.407174 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:01.407155    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ac739c6-6fe9-43a1-a92d-e1298aea7774-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:27:01.408992 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:01.408974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ac739c6-6fe9-43a1-a92d-e1298aea7774-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mxrqp\" (UID: \"5ac739c6-6fe9-43a1-a92d-e1298aea7774\") " pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:27:01.444505 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:01.444473    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gqjq9\""
Jul 28 01:27:01.453085 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:01.453047    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mxrqp"
Jul 28 01:27:01.590875 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:01.587600    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mxrqp"]
Jul 28 01:27:02.261966 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:02.261919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mxrqp" event={"ID":"5ac739c6-6fe9-43a1-a92d-e1298aea7774","Type":"ContainerStarted","Data":"cec4d4de479e91024b9a8bc7411edda6ba91b0173bd2fd9768270e73e04a1268"}
Jul 28 01:27:02.616160 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:02.616075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs\") pod \"network-metrics-daemon-dx5sm\" (UID: \"ddcb15a3-037d-45a0-8787-12585c0d2465\") " pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:27:02.616160 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:02.616130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret\") pod \"global-pull-secret-syncer-wpcdj\" (UID: \"cd1a9e35-4116-41f7-b275-9da51eb2549a\") " pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:27:02.619608 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:02.619580    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 01:27:02.620920 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:02.620901    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 01:27:02.630018 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:02.629984    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd1a9e35-4116-41f7-b275-9da51eb2549a-original-pull-secret\") pod \"global-pull-secret-syncer-wpcdj\" (UID: \"cd1a9e35-4116-41f7-b275-9da51eb2549a\") " pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:27:02.630173 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:02.629987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddcb15a3-037d-45a0-8787-12585c0d2465-metrics-certs\") pod \"network-metrics-daemon-dx5sm\" (UID: \"ddcb15a3-037d-45a0-8787-12585c0d2465\") " pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:27:02.692095 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:02.692052    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wpcdj"
Jul 28 01:27:02.716802 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:02.716768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j78fv\" (UniqueName: \"kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv\") pod \"network-check-target-zzc9f\" (UID: \"e5ba3944-6409-4af9-9bff-8eb91a65755a\") " pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:27:02.720201 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:02.720174    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 01:27:02.730976 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:02.730951    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 01:27:02.740780 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:02.740734    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j78fv\" (UniqueName: \"kubernetes.io/projected/e5ba3944-6409-4af9-9bff-8eb91a65755a-kube-api-access-j78fv\") pod \"network-check-target-zzc9f\" (UID: \"e5ba3944-6409-4af9-9bff-8eb91a65755a\") " pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:27:02.837470 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:02.837388    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wpcdj"]
Jul 28 01:27:02.841686 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:27:02.841659    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1a9e35_4116_41f7_b275_9da51eb2549a.slice/crio-6261481ea76fd178be26652a9af04100df1c1a19d2db29e83a79836558031067 WatchSource:0}: Error finding container 6261481ea76fd178be26652a9af04100df1c1a19d2db29e83a79836558031067: Status 404 returned error can't find the container with id 6261481ea76fd178be26652a9af04100df1c1a19d2db29e83a79836558031067
Jul 28 01:27:02.895661 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:02.895539    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5299b\""
Jul 28 01:27:02.903228 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:02.903199    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dx5sm"
Jul 28 01:27:03.001551 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:03.001516    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4mq45\""
Jul 28 01:27:03.009196 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:03.009171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:27:03.061604 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:03.061569    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dx5sm"]
Jul 28 01:27:03.067445 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:27:03.067393    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddcb15a3_037d_45a0_8787_12585c0d2465.slice/crio-d60edc25af99311af4f3d4c630cd9c400adf6bbbd162a1e7e5d9bff05db22d8f WatchSource:0}: Error finding container d60edc25af99311af4f3d4c630cd9c400adf6bbbd162a1e7e5d9bff05db22d8f: Status 404 returned error can't find the container with id d60edc25af99311af4f3d4c630cd9c400adf6bbbd162a1e7e5d9bff05db22d8f
Jul 28 01:27:03.120042 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:03.119976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls\") pod \"dns-default-t5gxc\" (UID: \"b6862e7b-5460-406a-b6dc-18de72afce4f\") " pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:27:03.123071 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:03.123041    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6862e7b-5460-406a-b6dc-18de72afce4f-metrics-tls\") pod \"dns-default-t5gxc\" (UID: \"b6862e7b-5460-406a-b6dc-18de72afce4f\") " pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:27:03.165128 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:03.164966    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zzc9f"]
Jul 28 01:27:03.168005 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:27:03.167974    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ba3944_6409_4af9_9bff_8eb91a65755a.slice/crio-81c1bb2b81b737e15343e964e049b4866b44ca3aca6c4cf16c35a8a3f5e6f759 WatchSource:0}: Error finding container 81c1bb2b81b737e15343e964e049b4866b44ca3aca6c4cf16c35a8a3f5e6f759: Status 404 returned error can't find the container with id 81c1bb2b81b737e15343e964e049b4866b44ca3aca6c4cf16c35a8a3f5e6f759
Jul 28 01:27:03.266700 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:03.266656    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zzc9f" event={"ID":"e5ba3944-6409-4af9-9bff-8eb91a65755a","Type":"ContainerStarted","Data":"81c1bb2b81b737e15343e964e049b4866b44ca3aca6c4cf16c35a8a3f5e6f759"}
Jul 28 01:27:03.267914 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:03.267882    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dx5sm" event={"ID":"ddcb15a3-037d-45a0-8787-12585c0d2465","Type":"ContainerStarted","Data":"d60edc25af99311af4f3d4c630cd9c400adf6bbbd162a1e7e5d9bff05db22d8f"}
Jul 28 01:27:03.269163 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:03.269137    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wpcdj" event={"ID":"cd1a9e35-4116-41f7-b275-9da51eb2549a","Type":"ContainerStarted","Data":"6261481ea76fd178be26652a9af04100df1c1a19d2db29e83a79836558031067"}
Jul 28 01:27:03.379774 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:03.379607    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mh7n2\""
Jul 28 01:27:03.387540 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:03.387418    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:27:03.563016 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:03.562980    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t5gxc"]
Jul 28 01:27:03.568417 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:27:03.568380    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6862e7b_5460_406a_b6dc_18de72afce4f.slice/crio-ed1bebb99d38c3a8b79c7c647ed0b848d6e1f718b62a8edfb3acb15539c18014 WatchSource:0}: Error finding container ed1bebb99d38c3a8b79c7c647ed0b848d6e1f718b62a8edfb3acb15539c18014: Status 404 returned error can't find the container with id ed1bebb99d38c3a8b79c7c647ed0b848d6e1f718b62a8edfb3acb15539c18014
Jul 28 01:27:04.274965 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:04.274924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t5gxc" event={"ID":"b6862e7b-5460-406a-b6dc-18de72afce4f","Type":"ContainerStarted","Data":"ed1bebb99d38c3a8b79c7c647ed0b848d6e1f718b62a8edfb3acb15539c18014"}
Jul 28 01:27:09.680113 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.680075    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67568f54f5-5j7d8"]
Jul 28 01:27:09.682507 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.682479    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.693038 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.692564    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 01:27:09.693038 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.692662    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 01:27:09.693038 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.692873    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-knzvx\""
Jul 28 01:27:09.693903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.693883    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 01:27:09.707492 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.707457    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 01:27:09.728306 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.728271    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"]
Jul 28 01:27:09.731445 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.731417    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:09.731600 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.731574    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67568f54f5-5j7d8"]
Jul 28 01:27:09.743267 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.743241    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 01:27:09.744366 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.744325    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 01:27:09.744598 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.744583    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 01:27:09.744725 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.744705    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 01:27:09.744791 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.744771    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 01:27:09.745284 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.745266    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 01:27:09.745518 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.745502    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 01:27:09.765068 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.765033    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"]
Jul 28 01:27:09.766060 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.766032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ce01934-b3bd-44d0-a32a-c549863b0bca-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86c476f8c-6qc46\" (UID: \"4ce01934-b3bd-44d0-a32a-c549863b0bca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:09.766222 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.766070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da463aa1-5a95-457f-88db-21c0ae437d5d-installation-pull-secrets\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.766222 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.766106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da463aa1-5a95-457f-88db-21c0ae437d5d-image-registry-private-configuration\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.766222 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.766131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da463aa1-5a95-457f-88db-21c0ae437d5d-ca-trust-extracted\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.766222 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.766191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da463aa1-5a95-457f-88db-21c0ae437d5d-registry-certificates\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.766471 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.766224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da463aa1-5a95-457f-88db-21c0ae437d5d-trusted-ca\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.766471 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.766301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ce01934-b3bd-44d0-a32a-c549863b0bca-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86c476f8c-6qc46\" (UID: \"4ce01934-b3bd-44d0-a32a-c549863b0bca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:09.766471 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.766394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:27:09.766471 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.766420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpsbn\" (UniqueName: \"kubernetes.io/projected/4ce01934-b3bd-44d0-a32a-c549863b0bca-kube-api-access-mpsbn\") pod \"cluster-proxy-proxy-agent-86c476f8c-6qc46\" (UID: \"4ce01934-b3bd-44d0-a32a-c549863b0bca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:09.766471 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.766448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da463aa1-5a95-457f-88db-21c0ae437d5d-registry-tls\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.766700 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.766474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd5mj\" (UniqueName: \"kubernetes.io/projected/da463aa1-5a95-457f-88db-21c0ae437d5d-kube-api-access-wd5mj\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.766700 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.766505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4ce01934-b3bd-44d0-a32a-c549863b0bca-ca\") pod \"cluster-proxy-proxy-agent-86c476f8c-6qc46\" (UID: \"4ce01934-b3bd-44d0-a32a-c549863b0bca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:09.766700 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.766526    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4ce01934-b3bd-44d0-a32a-c549863b0bca-hub\") pod \"cluster-proxy-proxy-agent-86c476f8c-6qc46\" (UID: \"4ce01934-b3bd-44d0-a32a-c549863b0bca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:09.766700 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.766552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4ce01934-b3bd-44d0-a32a-c549863b0bca-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86c476f8c-6qc46\" (UID: \"4ce01934-b3bd-44d0-a32a-c549863b0bca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:09.766700 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.766629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da463aa1-5a95-457f-88db-21c0ae437d5d-bound-sa-token\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.767122 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.767097    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f8169c2-427b-49b7-a2cc-8e8844f0d80c-service-ca-bundle\") pod \"router-default-68787665fc-c2vr7\" (UID: \"3f8169c2-427b-49b7-a2cc-8e8844f0d80c\") " pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:27:09.861637 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.861601    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67568f54f5-5j7d8"]
Jul 28 01:27:09.861878 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:27:09.861854    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-wd5mj registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-67568f54f5-5j7d8" podUID="da463aa1-5a95-457f-88db-21c0ae437d5d"
Jul 28 01:27:09.867907 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.867875    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da463aa1-5a95-457f-88db-21c0ae437d5d-image-registry-private-configuration\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.867907 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.867913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da463aa1-5a95-457f-88db-21c0ae437d5d-ca-trust-extracted\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.868118 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.867936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da463aa1-5a95-457f-88db-21c0ae437d5d-registry-certificates\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.868118 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.867968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da463aa1-5a95-457f-88db-21c0ae437d5d-trusted-ca\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.868118 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.868001    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ce01934-b3bd-44d0-a32a-c549863b0bca-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86c476f8c-6qc46\" (UID: \"4ce01934-b3bd-44d0-a32a-c549863b0bca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:09.868372 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.868176    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpsbn\" (UniqueName: \"kubernetes.io/projected/4ce01934-b3bd-44d0-a32a-c549863b0bca-kube-api-access-mpsbn\") pod \"cluster-proxy-proxy-agent-86c476f8c-6qc46\" (UID: \"4ce01934-b3bd-44d0-a32a-c549863b0bca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:09.868372 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.868225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da463aa1-5a95-457f-88db-21c0ae437d5d-registry-tls\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.868372 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.868263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd5mj\" (UniqueName: \"kubernetes.io/projected/da463aa1-5a95-457f-88db-21c0ae437d5d-kube-api-access-wd5mj\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.868372 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.868301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4ce01934-b3bd-44d0-a32a-c549863b0bca-ca\") pod \"cluster-proxy-proxy-agent-86c476f8c-6qc46\" (UID: \"4ce01934-b3bd-44d0-a32a-c549863b0bca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:09.868372 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.868325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4ce01934-b3bd-44d0-a32a-c549863b0bca-hub\") pod \"cluster-proxy-proxy-agent-86c476f8c-6qc46\" (UID: \"4ce01934-b3bd-44d0-a32a-c549863b0bca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:09.868372 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.868369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4ce01934-b3bd-44d0-a32a-c549863b0bca-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86c476f8c-6qc46\" (UID: \"4ce01934-b3bd-44d0-a32a-c549863b0bca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:09.868655 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.868410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da463aa1-5a95-457f-88db-21c0ae437d5d-bound-sa-token\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.868655 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.868423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da463aa1-5a95-457f-88db-21c0ae437d5d-ca-trust-extracted\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.868655 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.868452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ce01934-b3bd-44d0-a32a-c549863b0bca-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86c476f8c-6qc46\" (UID: \"4ce01934-b3bd-44d0-a32a-c549863b0bca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:09.868655 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.868477    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da463aa1-5a95-457f-88db-21c0ae437d5d-installation-pull-secrets\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.868847 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.868786    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ce01934-b3bd-44d0-a32a-c549863b0bca-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86c476f8c-6qc46\" (UID: \"4ce01934-b3bd-44d0-a32a-c549863b0bca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:09.868970 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.868936    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da463aa1-5a95-457f-88db-21c0ae437d5d-registry-certificates\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.869036 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.868986    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da463aa1-5a95-457f-88db-21c0ae437d5d-trusted-ca\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.871385 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.871303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da463aa1-5a95-457f-88db-21c0ae437d5d-installation-pull-secrets\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.871999 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.871956    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da463aa1-5a95-457f-88db-21c0ae437d5d-image-registry-private-configuration\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.872296 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.872249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4ce01934-b3bd-44d0-a32a-c549863b0bca-ca\") pod \"cluster-proxy-proxy-agent-86c476f8c-6qc46\" (UID: \"4ce01934-b3bd-44d0-a32a-c549863b0bca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:09.873122 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.872667    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da463aa1-5a95-457f-88db-21c0ae437d5d-registry-tls\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.873122 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.872676    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4ce01934-b3bd-44d0-a32a-c549863b0bca-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86c476f8c-6qc46\" (UID: \"4ce01934-b3bd-44d0-a32a-c549863b0bca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:09.873277 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.873228    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ce01934-b3bd-44d0-a32a-c549863b0bca-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86c476f8c-6qc46\" (UID: \"4ce01934-b3bd-44d0-a32a-c549863b0bca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:09.873957 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.873932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4ce01934-b3bd-44d0-a32a-c549863b0bca-hub\") pod \"cluster-proxy-proxy-agent-86c476f8c-6qc46\" (UID: \"4ce01934-b3bd-44d0-a32a-c549863b0bca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:09.885055 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.885028    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpsbn\" (UniqueName: \"kubernetes.io/projected/4ce01934-b3bd-44d0-a32a-c549863b0bca-kube-api-access-mpsbn\") pod \"cluster-proxy-proxy-agent-86c476f8c-6qc46\" (UID: \"4ce01934-b3bd-44d0-a32a-c549863b0bca\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:09.885180 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.885136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd5mj\" (UniqueName: \"kubernetes.io/projected/da463aa1-5a95-457f-88db-21c0ae437d5d-kube-api-access-wd5mj\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:09.893115 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:09.893079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da463aa1-5a95-457f-88db-21c0ae437d5d-bound-sa-token\") pod \"image-registry-67568f54f5-5j7d8\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") " pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:10.008126 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.008041    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-pczwk\""
Jul 28 01:27:10.009042 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.009010    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:27:10.057528 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.057487    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"
Jul 28 01:27:10.299810 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.299777    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:10.305595 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.305573    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:10.371799 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.371765    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da463aa1-5a95-457f-88db-21c0ae437d5d-image-registry-private-configuration\") pod \"da463aa1-5a95-457f-88db-21c0ae437d5d\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") "
Jul 28 01:27:10.371976 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.371844    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da463aa1-5a95-457f-88db-21c0ae437d5d-ca-trust-extracted\") pod \"da463aa1-5a95-457f-88db-21c0ae437d5d\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") "
Jul 28 01:27:10.371976 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.371874    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da463aa1-5a95-457f-88db-21c0ae437d5d-registry-certificates\") pod \"da463aa1-5a95-457f-88db-21c0ae437d5d\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") "
Jul 28 01:27:10.371976 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.371902    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da463aa1-5a95-457f-88db-21c0ae437d5d-bound-sa-token\") pod \"da463aa1-5a95-457f-88db-21c0ae437d5d\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") "
Jul 28 01:27:10.371976 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.371942    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da463aa1-5a95-457f-88db-21c0ae437d5d-registry-tls\") pod \"da463aa1-5a95-457f-88db-21c0ae437d5d\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") "
Jul 28 01:27:10.372196 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.371978    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da463aa1-5a95-457f-88db-21c0ae437d5d-installation-pull-secrets\") pod \"da463aa1-5a95-457f-88db-21c0ae437d5d\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") "
Jul 28 01:27:10.372196 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.372035    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da463aa1-5a95-457f-88db-21c0ae437d5d-trusted-ca\") pod \"da463aa1-5a95-457f-88db-21c0ae437d5d\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") "
Jul 28 01:27:10.372196 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.372073    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wd5mj\" (UniqueName: \"kubernetes.io/projected/da463aa1-5a95-457f-88db-21c0ae437d5d-kube-api-access-wd5mj\") pod \"da463aa1-5a95-457f-88db-21c0ae437d5d\" (UID: \"da463aa1-5a95-457f-88db-21c0ae437d5d\") "
Jul 28 01:27:10.375138 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.373627    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da463aa1-5a95-457f-88db-21c0ae437d5d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "da463aa1-5a95-457f-88db-21c0ae437d5d" (UID: "da463aa1-5a95-457f-88db-21c0ae437d5d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:27:10.375138 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.374091    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da463aa1-5a95-457f-88db-21c0ae437d5d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "da463aa1-5a95-457f-88db-21c0ae437d5d" (UID: "da463aa1-5a95-457f-88db-21c0ae437d5d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:27:10.375138 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.374162    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da463aa1-5a95-457f-88db-21c0ae437d5d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "da463aa1-5a95-457f-88db-21c0ae437d5d" (UID: "da463aa1-5a95-457f-88db-21c0ae437d5d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:27:10.387696 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.387651    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da463aa1-5a95-457f-88db-21c0ae437d5d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "da463aa1-5a95-457f-88db-21c0ae437d5d" (UID: "da463aa1-5a95-457f-88db-21c0ae437d5d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:27:10.388682 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.388545    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da463aa1-5a95-457f-88db-21c0ae437d5d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "da463aa1-5a95-457f-88db-21c0ae437d5d" (UID: "da463aa1-5a95-457f-88db-21c0ae437d5d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:27:10.389739 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.389711    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da463aa1-5a95-457f-88db-21c0ae437d5d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "da463aa1-5a95-457f-88db-21c0ae437d5d" (UID: "da463aa1-5a95-457f-88db-21c0ae437d5d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:27:10.389865 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.389842    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da463aa1-5a95-457f-88db-21c0ae437d5d-kube-api-access-wd5mj" (OuterVolumeSpecName: "kube-api-access-wd5mj") pod "da463aa1-5a95-457f-88db-21c0ae437d5d" (UID: "da463aa1-5a95-457f-88db-21c0ae437d5d"). InnerVolumeSpecName "kube-api-access-wd5mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:27:10.391748 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.391718    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da463aa1-5a95-457f-88db-21c0ae437d5d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "da463aa1-5a95-457f-88db-21c0ae437d5d" (UID: "da463aa1-5a95-457f-88db-21c0ae437d5d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:27:10.473523 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.473117    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da463aa1-5a95-457f-88db-21c0ae437d5d-trusted-ca\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:27:10.473523 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.473146    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd5mj\" (UniqueName: \"kubernetes.io/projected/da463aa1-5a95-457f-88db-21c0ae437d5d-kube-api-access-wd5mj\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:27:10.473523 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.473160    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da463aa1-5a95-457f-88db-21c0ae437d5d-image-registry-private-configuration\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:27:10.473523 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.473172    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da463aa1-5a95-457f-88db-21c0ae437d5d-ca-trust-extracted\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:27:10.473523 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.473184    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da463aa1-5a95-457f-88db-21c0ae437d5d-registry-certificates\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:27:10.473523 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.473196    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da463aa1-5a95-457f-88db-21c0ae437d5d-bound-sa-token\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:27:10.473523 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.473208    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da463aa1-5a95-457f-88db-21c0ae437d5d-registry-tls\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:27:10.473523 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.473220    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da463aa1-5a95-457f-88db-21c0ae437d5d-installation-pull-secrets\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:27:10.511725 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.511699    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68787665fc-c2vr7"]
Jul 28 01:27:10.516017 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:27:10.515983    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8169c2_427b_49b7_a2cc_8e8844f0d80c.slice/crio-446eb884ed5b4f60eba6427e88f7f1216d3e7ebc16922998cfc53031b4e4d7fb WatchSource:0}: Error finding container 446eb884ed5b4f60eba6427e88f7f1216d3e7ebc16922998cfc53031b4e4d7fb: Status 404 returned error can't find the container with id 446eb884ed5b4f60eba6427e88f7f1216d3e7ebc16922998cfc53031b4e4d7fb
Jul 28 01:27:10.545760 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:10.540680    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46"]
Jul 28 01:27:10.545760 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:27:10.545075    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ce01934_b3bd_44d0_a32a_c549863b0bca.slice/crio-7b0121800764f0cd2b6c6fba0b69a8556ace45c38f6f21e8cb621f04780d6d47 WatchSource:0}: Error finding container 7b0121800764f0cd2b6c6fba0b69a8556ace45c38f6f21e8cb621f04780d6d47: Status 404 returned error can't find the container with id 7b0121800764f0cd2b6c6fba0b69a8556ace45c38f6f21e8cb621f04780d6d47
Jul 28 01:27:11.310221 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.310183    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zzc9f" event={"ID":"e5ba3944-6409-4af9-9bff-8eb91a65755a","Type":"ContainerStarted","Data":"d3a2dbf2e0d1e4fc6f23cd6a01cf0894b3729e620ac0efb6966d39f130bea377"}
Jul 28 01:27:11.310992 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.310962    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:27:11.316817 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.316785    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68787665fc-c2vr7" event={"ID":"3f8169c2-427b-49b7-a2cc-8e8844f0d80c","Type":"ContainerStarted","Data":"4536fefa5fd791f3abac2d4edaec65e7774c21d7bd222ba5a5c85106c06b12cb"}
Jul 28 01:27:11.316949 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.316839    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68787665fc-c2vr7" event={"ID":"3f8169c2-427b-49b7-a2cc-8e8844f0d80c","Type":"ContainerStarted","Data":"446eb884ed5b4f60eba6427e88f7f1216d3e7ebc16922998cfc53031b4e4d7fb"}
Jul 28 01:27:11.318968 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.318942    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46" event={"ID":"4ce01934-b3bd-44d0-a32a-c549863b0bca","Type":"ContainerStarted","Data":"7b0121800764f0cd2b6c6fba0b69a8556ace45c38f6f21e8cb621f04780d6d47"}
Jul 28 01:27:11.320987 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.320961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mxrqp" event={"ID":"5ac739c6-6fe9-43a1-a92d-e1298aea7774","Type":"ContainerStarted","Data":"bd5d04f0351ea81c5a992139cc415d709462b2a65e967e3a04d561a3c4f31abf"}
Jul 28 01:27:11.323790 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.323611    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dx5sm" event={"ID":"ddcb15a3-037d-45a0-8787-12585c0d2465","Type":"ContainerStarted","Data":"32da1f2892ae06d722f1ce0736b2d27aeb19ebfc83015236415057129f1f9445"}
Jul 28 01:27:11.323790 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.323656    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dx5sm" event={"ID":"ddcb15a3-037d-45a0-8787-12585c0d2465","Type":"ContainerStarted","Data":"e5452ce1d137d581468d0e30de427a02e30caeb95551984ed1cc5e252e05c4a6"}
Jul 28 01:27:11.325701 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.325678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wpcdj" event={"ID":"cd1a9e35-4116-41f7-b275-9da51eb2549a","Type":"ContainerStarted","Data":"b74ae92be3a6a4520e5f97294874d090431bd03ad3f9ac9c75657239f4a8eb4c"}
Jul 28 01:27:11.328434 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.328389    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zzc9f" podStartSLOduration=67.219661536 podStartE2EDuration="1m14.328373676s" totalImagesPullingTime="7.10871214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:11.327119163 +0000 UTC m=+74.904576781" watchObservedRunningTime="2026-07-28 01:27:11.328373676 +0000 UTC m=+74.905831271"
Jul 28 01:27:11.329284 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.329223    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t5gxc" event={"ID":"b6862e7b-5460-406a-b6dc-18de72afce4f","Type":"ContainerStarted","Data":"4c69ddc00901942d86a3efdab960ef1891612c2aafbba732e8f90dae5275f1d8"}
Jul 28 01:27:11.329284 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.329264    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67568f54f5-5j7d8"
Jul 28 01:27:11.329787 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.329271    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t5gxc" event={"ID":"b6862e7b-5460-406a-b6dc-18de72afce4f","Type":"ContainerStarted","Data":"c852881664b5165626f708b34cac0a9a48aae34a34472ce9661194155b587a7f"}
Jul 28 01:27:11.329787 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.329641    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:27:11.344782 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.344724    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dx5sm" podStartSLOduration=67.136701822 podStartE2EDuration="1m14.344699983s" totalImagesPullingTime="7.207998161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:11.344346311 +0000 UTC m=+74.921803927" watchObservedRunningTime="2026-07-28 01:27:11.344699983 +0000 UTC m=+74.922157605"
Jul 28 01:27:11.364565 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.364496    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wpcdj" podStartSLOduration=66.89821903 podStartE2EDuration="1m14.364470911s" totalImagesPullingTime="7.466251881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:11.362473395 +0000 UTC m=+74.939931013" watchObservedRunningTime="2026-07-28 01:27:11.364470911 +0000 UTC m=+74.941928521"
Jul 28 01:27:11.390519 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.390461    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-68787665fc-c2vr7" podStartSLOduration=34.390437826 podStartE2EDuration="34.390437826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:11.389128493 +0000 UTC m=+74.966586111" watchObservedRunningTime="2026-07-28 01:27:11.390437826 +0000 UTC m=+74.967895448"
Jul 28 01:27:11.433665 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.432722    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-t5gxc" podStartSLOduration=33.676638945 podStartE2EDuration="40.432695511s" totalImagesPullingTime="6.756056566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:11.406636465 +0000 UTC m=+74.984094083" watchObservedRunningTime="2026-07-28 01:27:11.432695511 +0000 UTC m=+75.010153129"
Jul 28 01:27:11.444095 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.444030    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67568f54f5-5j7d8"]
Jul 28 01:27:11.446697 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:11.446665    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67568f54f5-5j7d8"]
Jul 28 01:27:12.009554 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:12.009502    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:27:12.012985 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:12.012940    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:27:12.334501 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:12.334413    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:27:12.335504 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:12.335476    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-68787665fc-c2vr7"
Jul 28 01:27:12.985289 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:12.985250    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da463aa1-5a95-457f-88db-21c0ae437d5d" path="/var/lib/kubelet/pods/da463aa1-5a95-457f-88db-21c0ae437d5d/volumes"
Jul 28 01:27:16.348806 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:16.348771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46" event={"ID":"4ce01934-b3bd-44d0-a32a-c549863b0bca","Type":"ContainerStarted","Data":"3ff8a27fdedf60e43dd8b588baf26433c8a1a7a5e20380315c835b5fd6734db0"}
Jul 28 01:27:16.349303 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:16.348819    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46" event={"ID":"4ce01934-b3bd-44d0-a32a-c549863b0bca","Type":"ContainerStarted","Data":"84e641bdd3d1a0045dbe7a8e56cbc36f9bab8a44088eb08348a8ac39a7710603"}
Jul 28 01:27:16.349303 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:16.348835    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46" event={"ID":"4ce01934-b3bd-44d0-a32a-c549863b0bca","Type":"ContainerStarted","Data":"d0b804202b9ecd82f677805ab8785d753873bb65e34627d2ca19f2a7433a9047"}
Jul 28 01:27:16.350998 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:16.350967    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mxrqp" event={"ID":"5ac739c6-6fe9-43a1-a92d-e1298aea7774","Type":"ContainerStarted","Data":"13d8998258eeed260898765ac01e31e074e144248d56e9ef399ab81707c070cc"}
Jul 28 01:27:16.374146 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:16.374090    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86c476f8c-6qc46" podStartSLOduration=2.067174731 podStartE2EDuration="7.374073269s" totalImagesPullingTime="5.306898538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:16.372681785 +0000 UTC m=+79.950139404" watchObservedRunningTime="2026-07-28 01:27:16.374073269 +0000 UTC m=+79.951530885"
Jul 28 01:27:20.366931 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:20.366896    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mxrqp" event={"ID":"5ac739c6-6fe9-43a1-a92d-e1298aea7774","Type":"ContainerStarted","Data":"8c9e036d32e5f4c1b15938f11431c3bf14497ded29f3c6e39b352be7aeadc723"}
Jul 28 01:27:20.386683 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:20.386630    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mxrqp" podStartSLOduration=17.71906905 podStartE2EDuration="35.386613221s" totalImagesPullingTime="17.667544171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:20.386090378 +0000 UTC m=+83.963547996" watchObservedRunningTime="2026-07-28 01:27:20.386613221 +0000 UTC m=+83.964070841"
Jul 28 01:27:21.335400 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:21.335370    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t5gxc"
Jul 28 01:27:34.439457 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.439423    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jlpsc"]
Jul 28 01:27:34.444719 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.444699    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.451315 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.451091    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 01:27:34.465036 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:27:34.465006    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-dockercfg-vt9th\" is forbidden: User \"system:node:ip-10-0-139-81.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-139-81.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vt9th\"" type="*v1.Secret"
Jul 28 01:27:34.465036 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:27:34.465012    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"node-exporter-jlpsc\" is forbidden: User \"system:node:ip-10-0-139-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-139-81.ec2.internal' and this object" podUID="fd3e8c04-97b7-4b06-9fe9-d889ac1202ce" pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.465036 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:27:34.465030    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-tls\" is forbidden: User \"system:node:ip-10-0-139-81.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-139-81.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret"
Jul 28 01:27:34.465287 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:27:34.465009    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"node-exporter-accelerators-collector-config\" is forbidden: User \"system:node:ip-10-0-139-81.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-139-81.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap"
Jul 28 01:27:34.465287 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:27:34.465163    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"metrics-client-ca\" is forbidden: User \"system:node:ip-10-0-139-81.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-139-81.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap"
Jul 28 01:27:34.465287 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:27:34.465225    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-139-81.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-139-81.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 28 01:27:34.465287 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:27:34.465247    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-139-81.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-139-81.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 28 01:27:34.553737 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.553703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-root\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.553880 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.553748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-textfile\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.553880 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.553793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-metrics-client-ca\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.553880 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.553860    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-tls\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.553880 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.553877    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-wtmp\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.554036 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.553897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-accelerators-collector-config\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.554036 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.553990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-sys\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.554036 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.554020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.554144 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.554038    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79cqb\" (UniqueName: \"kubernetes.io/projected/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-kube-api-access-79cqb\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.654712 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.654683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-root\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.654868 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.654725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-textfile\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.654868 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.654754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-metrics-client-ca\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.654868 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.654788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-tls\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.654868 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.654804    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-root\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.654868 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.654817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-wtmp\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.654868 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.654855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-accelerators-collector-config\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.655110 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.654914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-sys\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.655110 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.654936    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-wtmp\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.655110 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.654946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.655110 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.654973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79cqb\" (UniqueName: \"kubernetes.io/projected/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-kube-api-access-79cqb\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.655110 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.654970    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-sys\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.655110 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.655074    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-textfile\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:34.657243 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:34.657227    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:35.589787 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:35.589751    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 01:27:35.655001 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:27:35.654971    2578 configmap.go:193] Couldn't get configMap openshift-monitoring/metrics-client-ca: failed to sync configmap cache: timed out waiting for the condition
Jul 28 01:27:35.655162 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:27:35.655039    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-metrics-client-ca podName:fd3e8c04-97b7-4b06-9fe9-d889ac1202ce nodeName:}" failed. No retries permitted until 2026-07-28 01:27:36.155022759 +0000 UTC m=+99.732480357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-client-ca" (UniqueName: "kubernetes.io/configmap/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-metrics-client-ca") pod "node-exporter-jlpsc" (UID: "fd3e8c04-97b7-4b06-9fe9-d889ac1202ce") : failed to sync configmap cache: timed out waiting for the condition
Jul 28 01:27:35.655162 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:27:35.654970    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: failed to sync secret cache: timed out waiting for the condition
Jul 28 01:27:35.655162 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:27:35.655115    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-tls podName:fd3e8c04-97b7-4b06-9fe9-d889ac1202ce nodeName:}" failed. No retries permitted until 2026-07-28 01:27:36.155100877 +0000 UTC m=+99.732558475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-tls") pod "node-exporter-jlpsc" (UID: "fd3e8c04-97b7-4b06-9fe9-d889ac1202ce") : failed to sync secret cache: timed out waiting for the condition
Jul 28 01:27:35.655162 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:27:35.655113    2578 configmap.go:193] Couldn't get configMap openshift-monitoring/node-exporter-accelerators-collector-config: failed to sync configmap cache: timed out waiting for the condition
Jul 28 01:27:35.655358 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:27:35.655192    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-accelerators-collector-config podName:fd3e8c04-97b7-4b06-9fe9-d889ac1202ce nodeName:}" failed. No retries permitted until 2026-07-28 01:27:36.155176647 +0000 UTC m=+99.732634249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-accelerators-collector-config" (UniqueName: "kubernetes.io/configmap/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-accelerators-collector-config") pod "node-exporter-jlpsc" (UID: "fd3e8c04-97b7-4b06-9fe9-d889ac1202ce") : failed to sync configmap cache: timed out waiting for the condition
Jul 28 01:27:35.664727 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:35.664709    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 01:27:35.944481 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:35.944406    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 01:27:35.959237 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:35.959212    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 01:27:35.966735 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:35.966713    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79cqb\" (UniqueName: \"kubernetes.io/projected/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-kube-api-access-79cqb\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:35.977341 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:35.977305    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 01:27:35.998688 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:35.998660    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vt9th\""
Jul 28 01:27:36.165166 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:36.165117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-metrics-client-ca\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:36.165403 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:36.165178    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-tls\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:36.165403 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:36.165212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-accelerators-collector-config\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:36.165728 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:36.165704    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-metrics-client-ca\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:36.165868 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:36.165847    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-accelerators-collector-config\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:36.167486 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:36.167468    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd3e8c04-97b7-4b06-9fe9-d889ac1202ce-node-exporter-tls\") pod \"node-exporter-jlpsc\" (UID: \"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce\") " pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:36.253881 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:36.253792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jlpsc"
Jul 28 01:27:36.264916 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:27:36.264887    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3e8c04_97b7_4b06_9fe9_d889ac1202ce.slice/crio-a305ef4d34f9b1c297199b8b7e1fbb77c5b55b51ffbd6bda6e0b47d54dac0707 WatchSource:0}: Error finding container a305ef4d34f9b1c297199b8b7e1fbb77c5b55b51ffbd6bda6e0b47d54dac0707: Status 404 returned error can't find the container with id a305ef4d34f9b1c297199b8b7e1fbb77c5b55b51ffbd6bda6e0b47d54dac0707
Jul 28 01:27:36.419262 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:36.419229    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jlpsc" event={"ID":"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce","Type":"ContainerStarted","Data":"a305ef4d34f9b1c297199b8b7e1fbb77c5b55b51ffbd6bda6e0b47d54dac0707"}
Jul 28 01:27:37.424398 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:37.424291    2578 generic.go:356] "Generic (PLEG): container finished" podID="fd3e8c04-97b7-4b06-9fe9-d889ac1202ce" containerID="24ebfb17c37ab289eeb52a980eb8c256e46c02b461ab5ababdacf99f235b7609" exitCode=0
Jul 28 01:27:37.424398 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:37.424370    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jlpsc" event={"ID":"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce","Type":"ContainerDied","Data":"24ebfb17c37ab289eeb52a980eb8c256e46c02b461ab5ababdacf99f235b7609"}
Jul 28 01:27:38.429803 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:38.429767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jlpsc" event={"ID":"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce","Type":"ContainerStarted","Data":"016dde6cac9629c6573f4bff9ba4890add051316de29f03786a60c65b64b9499"}
Jul 28 01:27:38.430243 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:38.429828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jlpsc" event={"ID":"fd3e8c04-97b7-4b06-9fe9-d889ac1202ce","Type":"ContainerStarted","Data":"d9b5f4c59e105aea2c752251325a8120ded3072b415c4d8a4e88587ce32649fa"}
Jul 28 01:27:38.453935 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:38.453887    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jlpsc" podStartSLOduration=3.546226202 podStartE2EDuration="4.453872987s" totalImagesPullingTime="844.777492ms" totalInitContainerRuntime="62.869293ms" isStatefulPod=true podCreationTimestamp="2026-07-28 01:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:38.451610408 +0000 UTC m=+102.029068026" watchObservedRunningTime="2026-07-28 01:27:38.453872987 +0000 UTC m=+102.031330603"
Jul 28 01:27:43.339366 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:43.339319    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zzc9f"
Jul 28 01:27:51.476845 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:51.476803    2578 generic.go:356] "Generic (PLEG): container finished" podID="79ec8bac-483f-4773-a8b0-5ac56d23c8bb" containerID="66332cdfe0a8f966c0fd57692eccf8bac07309b26cf9b834b533c8cdde257aa6" exitCode=0
Jul 28 01:27:51.477301 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:51.476883    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2" event={"ID":"79ec8bac-483f-4773-a8b0-5ac56d23c8bb","Type":"ContainerDied","Data":"66332cdfe0a8f966c0fd57692eccf8bac07309b26cf9b834b533c8cdde257aa6"}
Jul 28 01:27:51.477301 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:51.477218    2578 scope.go:117] "RemoveContainer" containerID="66332cdfe0a8f966c0fd57692eccf8bac07309b26cf9b834b533c8cdde257aa6"
Jul 28 01:27:52.482382 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:27:52.482326    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2" event={"ID":"79ec8bac-483f-4773-a8b0-5ac56d23c8bb","Type":"ContainerStarted","Data":"88d30cdf011feec54b9fa183a80dce0ede893a5654c08989e37c1bad7ea21fd0"}
Jul 28 01:30:56.824514 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:30:56.824483    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 01:31:03.151012 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.150972    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-5mh9h"]
Jul 28 01:31:03.153221 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.153206    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-5mh9h"
Jul 28 01:31:03.156076 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.156055    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 28 01:31:03.156297 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.156280    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 01:31:03.156516 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.156493    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 01:31:03.156597 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.156554    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 01:31:03.157628 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.157610    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 01:31:03.157886 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.157867    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-vtwrz\""
Jul 28 01:31:03.166606 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.166583    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-5mh9h"]
Jul 28 01:31:03.222123 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.222084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/95d130f8-30f9-4b82-92a7-fa2b97a9d7ef-cabundle0\") pod \"keda-operator-77f648d694-5mh9h\" (UID: \"95d130f8-30f9-4b82-92a7-fa2b97a9d7ef\") " pod="openshift-keda/keda-operator-77f648d694-5mh9h"
Jul 28 01:31:03.222312 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.222137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/95d130f8-30f9-4b82-92a7-fa2b97a9d7ef-certificates\") pod \"keda-operator-77f648d694-5mh9h\" (UID: \"95d130f8-30f9-4b82-92a7-fa2b97a9d7ef\") " pod="openshift-keda/keda-operator-77f648d694-5mh9h"
Jul 28 01:31:03.222312 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.222187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s89f\" (UniqueName: \"kubernetes.io/projected/95d130f8-30f9-4b82-92a7-fa2b97a9d7ef-kube-api-access-6s89f\") pod \"keda-operator-77f648d694-5mh9h\" (UID: \"95d130f8-30f9-4b82-92a7-fa2b97a9d7ef\") " pod="openshift-keda/keda-operator-77f648d694-5mh9h"
Jul 28 01:31:03.323232 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.323199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/95d130f8-30f9-4b82-92a7-fa2b97a9d7ef-certificates\") pod \"keda-operator-77f648d694-5mh9h\" (UID: \"95d130f8-30f9-4b82-92a7-fa2b97a9d7ef\") " pod="openshift-keda/keda-operator-77f648d694-5mh9h"
Jul 28 01:31:03.323446 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.323238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s89f\" (UniqueName: \"kubernetes.io/projected/95d130f8-30f9-4b82-92a7-fa2b97a9d7ef-kube-api-access-6s89f\") pod \"keda-operator-77f648d694-5mh9h\" (UID: \"95d130f8-30f9-4b82-92a7-fa2b97a9d7ef\") " pod="openshift-keda/keda-operator-77f648d694-5mh9h"
Jul 28 01:31:03.323446 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.323300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/95d130f8-30f9-4b82-92a7-fa2b97a9d7ef-cabundle0\") pod \"keda-operator-77f648d694-5mh9h\" (UID: \"95d130f8-30f9-4b82-92a7-fa2b97a9d7ef\") " pod="openshift-keda/keda-operator-77f648d694-5mh9h"
Jul 28 01:31:03.323446 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:31:03.323387    2578 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 28 01:31:03.323446 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:31:03.323410    2578 secret.go:281] references non-existent secret key: ca.crt
Jul 28 01:31:03.323446 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:31:03.323421    2578 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 01:31:03.323446 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:31:03.323435    2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-5mh9h: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 01:31:03.323766 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:31:03.323488    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95d130f8-30f9-4b82-92a7-fa2b97a9d7ef-certificates podName:95d130f8-30f9-4b82-92a7-fa2b97a9d7ef nodeName:}" failed. No retries permitted until 2026-07-28 01:31:03.823473121 +0000 UTC m=+307.400930737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/95d130f8-30f9-4b82-92a7-fa2b97a9d7ef-certificates") pod "keda-operator-77f648d694-5mh9h" (UID: "95d130f8-30f9-4b82-92a7-fa2b97a9d7ef") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 01:31:03.324043 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.324020    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/95d130f8-30f9-4b82-92a7-fa2b97a9d7ef-cabundle0\") pod \"keda-operator-77f648d694-5mh9h\" (UID: \"95d130f8-30f9-4b82-92a7-fa2b97a9d7ef\") " pod="openshift-keda/keda-operator-77f648d694-5mh9h"
Jul 28 01:31:03.345709 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.345679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s89f\" (UniqueName: \"kubernetes.io/projected/95d130f8-30f9-4b82-92a7-fa2b97a9d7ef-kube-api-access-6s89f\") pod \"keda-operator-77f648d694-5mh9h\" (UID: \"95d130f8-30f9-4b82-92a7-fa2b97a9d7ef\") " pod="openshift-keda/keda-operator-77f648d694-5mh9h"
Jul 28 01:31:03.827397 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.827357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/95d130f8-30f9-4b82-92a7-fa2b97a9d7ef-certificates\") pod \"keda-operator-77f648d694-5mh9h\" (UID: \"95d130f8-30f9-4b82-92a7-fa2b97a9d7ef\") " pod="openshift-keda/keda-operator-77f648d694-5mh9h"
Jul 28 01:31:03.829795 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.829773    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/95d130f8-30f9-4b82-92a7-fa2b97a9d7ef-certificates\") pod \"keda-operator-77f648d694-5mh9h\" (UID: \"95d130f8-30f9-4b82-92a7-fa2b97a9d7ef\") " pod="openshift-keda/keda-operator-77f648d694-5mh9h"
Jul 28 01:31:03.836081 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.836057    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-ttzr7"]
Jul 28 01:31:03.838304 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.838288    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-ttzr7"
Jul 28 01:31:03.841154 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.841132    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 28 01:31:03.850549 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.850526    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-ttzr7"]
Jul 28 01:31:03.928238 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.928207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdlfw\" (UniqueName: \"kubernetes.io/projected/b340f99d-e927-4641-add9-bc781ed7cff4-kube-api-access-cdlfw\") pod \"keda-admission-56f9798bb4-ttzr7\" (UID: \"b340f99d-e927-4641-add9-bc781ed7cff4\") " pod="openshift-keda/keda-admission-56f9798bb4-ttzr7"
Jul 28 01:31:03.928431 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:03.928275    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b340f99d-e927-4641-add9-bc781ed7cff4-certificates\") pod \"keda-admission-56f9798bb4-ttzr7\" (UID: \"b340f99d-e927-4641-add9-bc781ed7cff4\") " pod="openshift-keda/keda-admission-56f9798bb4-ttzr7"
Jul 28 01:31:04.029585 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:04.029552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdlfw\" (UniqueName: \"kubernetes.io/projected/b340f99d-e927-4641-add9-bc781ed7cff4-kube-api-access-cdlfw\") pod \"keda-admission-56f9798bb4-ttzr7\" (UID: \"b340f99d-e927-4641-add9-bc781ed7cff4\") " pod="openshift-keda/keda-admission-56f9798bb4-ttzr7"
Jul 28 01:31:04.029585 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:04.029589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b340f99d-e927-4641-add9-bc781ed7cff4-certificates\") pod \"keda-admission-56f9798bb4-ttzr7\" (UID: \"b340f99d-e927-4641-add9-bc781ed7cff4\") " pod="openshift-keda/keda-admission-56f9798bb4-ttzr7"
Jul 28 01:31:04.029850 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:31:04.029720    2578 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 28 01:31:04.029850 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:31:04.029747    2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-56f9798bb4-ttzr7: secret "keda-admission-webhooks-certs" not found
Jul 28 01:31:04.029850 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:31:04.029801    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b340f99d-e927-4641-add9-bc781ed7cff4-certificates podName:b340f99d-e927-4641-add9-bc781ed7cff4 nodeName:}" failed. No retries permitted until 2026-07-28 01:31:04.529782496 +0000 UTC m=+308.107240093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b340f99d-e927-4641-add9-bc781ed7cff4-certificates") pod "keda-admission-56f9798bb4-ttzr7" (UID: "b340f99d-e927-4641-add9-bc781ed7cff4") : secret "keda-admission-webhooks-certs" not found
Jul 28 01:31:04.041755 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:04.041730    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdlfw\" (UniqueName: \"kubernetes.io/projected/b340f99d-e927-4641-add9-bc781ed7cff4-kube-api-access-cdlfw\") pod \"keda-admission-56f9798bb4-ttzr7\" (UID: \"b340f99d-e927-4641-add9-bc781ed7cff4\") " pod="openshift-keda/keda-admission-56f9798bb4-ttzr7"
Jul 28 01:31:04.063223 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:04.063193    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-5mh9h"
Jul 28 01:31:04.187057 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:04.186951    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-5mh9h"]
Jul 28 01:31:04.189679 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:31:04.189651    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d130f8_30f9_4b82_92a7_fa2b97a9d7ef.slice/crio-be98310eeb188416d8d1585afc08811761b96985c0223955abdcf48b141ff40f WatchSource:0}: Error finding container be98310eeb188416d8d1585afc08811761b96985c0223955abdcf48b141ff40f: Status 404 returned error can't find the container with id be98310eeb188416d8d1585afc08811761b96985c0223955abdcf48b141ff40f
Jul 28 01:31:04.190942 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:04.190927    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:31:04.534360 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:04.534314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b340f99d-e927-4641-add9-bc781ed7cff4-certificates\") pod \"keda-admission-56f9798bb4-ttzr7\" (UID: \"b340f99d-e927-4641-add9-bc781ed7cff4\") " pod="openshift-keda/keda-admission-56f9798bb4-ttzr7"
Jul 28 01:31:04.536713 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:04.536687    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b340f99d-e927-4641-add9-bc781ed7cff4-certificates\") pod \"keda-admission-56f9798bb4-ttzr7\" (UID: \"b340f99d-e927-4641-add9-bc781ed7cff4\") " pod="openshift-keda/keda-admission-56f9798bb4-ttzr7"
Jul 28 01:31:04.747895 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:04.747853    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-ttzr7"
Jul 28 01:31:04.884732 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:04.884705    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-ttzr7"]
Jul 28 01:31:04.888705 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:31:04.888675    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb340f99d_e927_4641_add9_bc781ed7cff4.slice/crio-a88a3228abcc6306e87686d13e2c4aec1b1f76ef8cfd3b16a2d25f7694ec38b0 WatchSource:0}: Error finding container a88a3228abcc6306e87686d13e2c4aec1b1f76ef8cfd3b16a2d25f7694ec38b0: Status 404 returned error can't find the container with id a88a3228abcc6306e87686d13e2c4aec1b1f76ef8cfd3b16a2d25f7694ec38b0
Jul 28 01:31:05.117760 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:05.117671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-ttzr7" event={"ID":"b340f99d-e927-4641-add9-bc781ed7cff4","Type":"ContainerStarted","Data":"a88a3228abcc6306e87686d13e2c4aec1b1f76ef8cfd3b16a2d25f7694ec38b0"}
Jul 28 01:31:05.119173 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:05.119144    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-5mh9h" event={"ID":"95d130f8-30f9-4b82-92a7-fa2b97a9d7ef","Type":"ContainerStarted","Data":"be98310eeb188416d8d1585afc08811761b96985c0223955abdcf48b141ff40f"}
Jul 28 01:31:08.133671 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:08.133632    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-ttzr7" event={"ID":"b340f99d-e927-4641-add9-bc781ed7cff4","Type":"ContainerStarted","Data":"051d8c90d5731db157279737773e3864512804539f33beb718a8dcfaff30a1d9"}
Jul 28 01:31:08.134135 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:08.133761    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-ttzr7"
Jul 28 01:31:08.135281 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:08.135255    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-5mh9h" event={"ID":"95d130f8-30f9-4b82-92a7-fa2b97a9d7ef","Type":"ContainerStarted","Data":"6cd771c34f15c61537722d258a116713a26ed8aa1302baf86e2408c90ea1e208"}
Jul 28 01:31:08.135472 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:08.135305    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-5mh9h"
Jul 28 01:31:08.151356 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:08.151280    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-ttzr7" podStartSLOduration=2.0996768 podStartE2EDuration="5.15126355s" totalImagesPullingTime="3.05158675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:31:08.150139339 +0000 UTC m=+311.727596955" watchObservedRunningTime="2026-07-28 01:31:08.15126355 +0000 UTC m=+311.728721145"
Jul 28 01:31:08.168190 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:08.168142    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-5mh9h" podStartSLOduration=1.4071459800000001 podStartE2EDuration="5.168127249s" totalImagesPullingTime="3.760981269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:31:08.167049042 +0000 UTC m=+311.744506677" watchObservedRunningTime="2026-07-28 01:31:08.168127249 +0000 UTC m=+311.745584867"
Jul 28 01:31:29.142204 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:29.142170    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-ttzr7"
Jul 28 01:31:29.142754 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:31:29.142240    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-5mh9h"
Jul 28 01:32:21.497702 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:21.497674    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-rtfzg"]
Jul 28 01:32:21.499860 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:21.499841    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-rtfzg"
Jul 28 01:32:21.503349 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:21.503317    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dp2vv\""
Jul 28 01:32:21.503479 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:21.503374    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 01:32:21.503479 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:21.503432    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 01:32:21.510397 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:21.510036    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 01:32:21.513004 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:21.512969    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-rtfzg"]
Jul 28 01:32:21.594724 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:21.594685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/40c6c649-9390-475e-a84e-b95b541fb6f0-data\") pod \"seaweedfs-64568bcd49-rtfzg\" (UID: \"40c6c649-9390-475e-a84e-b95b541fb6f0\") " pod="kserve/seaweedfs-64568bcd49-rtfzg"
Jul 28 01:32:21.594905 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:21.594752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptfjw\" (UniqueName: \"kubernetes.io/projected/40c6c649-9390-475e-a84e-b95b541fb6f0-kube-api-access-ptfjw\") pod \"seaweedfs-64568bcd49-rtfzg\" (UID: \"40c6c649-9390-475e-a84e-b95b541fb6f0\") " pod="kserve/seaweedfs-64568bcd49-rtfzg"
Jul 28 01:32:21.695135 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:21.695097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptfjw\" (UniqueName: \"kubernetes.io/projected/40c6c649-9390-475e-a84e-b95b541fb6f0-kube-api-access-ptfjw\") pod \"seaweedfs-64568bcd49-rtfzg\" (UID: \"40c6c649-9390-475e-a84e-b95b541fb6f0\") " pod="kserve/seaweedfs-64568bcd49-rtfzg"
Jul 28 01:32:21.695268 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:21.695147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/40c6c649-9390-475e-a84e-b95b541fb6f0-data\") pod \"seaweedfs-64568bcd49-rtfzg\" (UID: \"40c6c649-9390-475e-a84e-b95b541fb6f0\") " pod="kserve/seaweedfs-64568bcd49-rtfzg"
Jul 28 01:32:21.695528 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:21.695512    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/40c6c649-9390-475e-a84e-b95b541fb6f0-data\") pod \"seaweedfs-64568bcd49-rtfzg\" (UID: \"40c6c649-9390-475e-a84e-b95b541fb6f0\") " pod="kserve/seaweedfs-64568bcd49-rtfzg"
Jul 28 01:32:21.703737 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:21.703705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptfjw\" (UniqueName: \"kubernetes.io/projected/40c6c649-9390-475e-a84e-b95b541fb6f0-kube-api-access-ptfjw\") pod \"seaweedfs-64568bcd49-rtfzg\" (UID: \"40c6c649-9390-475e-a84e-b95b541fb6f0\") " pod="kserve/seaweedfs-64568bcd49-rtfzg"
Jul 28 01:32:21.814031 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:21.813997    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-rtfzg"
Jul 28 01:32:21.945520 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:21.945487    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-rtfzg"]
Jul 28 01:32:21.948974 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:32:21.948949    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c6c649_9390_475e_a84e_b95b541fb6f0.slice/crio-dbf3606aafee8b36bc99dd36aae7b11d16d401ca6b9d74e2b1726ec344bbdc7e WatchSource:0}: Error finding container dbf3606aafee8b36bc99dd36aae7b11d16d401ca6b9d74e2b1726ec344bbdc7e: Status 404 returned error can't find the container with id dbf3606aafee8b36bc99dd36aae7b11d16d401ca6b9d74e2b1726ec344bbdc7e
Jul 28 01:32:22.392216 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:22.392177    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-rtfzg" event={"ID":"40c6c649-9390-475e-a84e-b95b541fb6f0","Type":"ContainerStarted","Data":"dbf3606aafee8b36bc99dd36aae7b11d16d401ca6b9d74e2b1726ec344bbdc7e"}
Jul 28 01:32:26.409019 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:26.408984    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-rtfzg" event={"ID":"40c6c649-9390-475e-a84e-b95b541fb6f0","Type":"ContainerStarted","Data":"56edd77c923e7259511dcb72b9d881f85a633e2559f2eefb37c025a8d70e764c"}
Jul 28 01:32:26.409530 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:26.409087    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-rtfzg"
Jul 28 01:32:32.414540 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:32.414504    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-rtfzg"
Jul 28 01:32:32.432510 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:32:32.432462    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-rtfzg" podStartSLOduration=7.949091637 podStartE2EDuration="11.43244582s" totalImagesPullingTime="3.483354183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:32:26.445663196 +0000 UTC m=+390.023120812" watchObservedRunningTime="2026-07-28 01:32:32.43244582 +0000 UTC m=+396.009903437"
Jul 28 01:33:18.565142 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:18.565061    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qt55f"]
Jul 28 01:33:18.568257 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:18.568239    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qt55f"
Jul 28 01:33:18.571136 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:18.571111    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 28 01:33:18.571136 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:18.571115    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:33:18.571316 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:18.571217    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-s5pxm\""
Jul 28 01:33:18.582193 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:18.582170    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qt55f"]
Jul 28 01:33:18.707703 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:18.707668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mhln\" (UniqueName: \"kubernetes.io/projected/a083ee12-bdd8-40a8-bf20-c009e0c8f366-kube-api-access-9mhln\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qt55f\" (UID: \"a083ee12-bdd8-40a8-bf20-c009e0c8f366\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qt55f"
Jul 28 01:33:18.707891 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:18.707740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a083ee12-bdd8-40a8-bf20-c009e0c8f366-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qt55f\" (UID: \"a083ee12-bdd8-40a8-bf20-c009e0c8f366\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qt55f"
Jul 28 01:33:18.808090 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:18.808052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a083ee12-bdd8-40a8-bf20-c009e0c8f366-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qt55f\" (UID: \"a083ee12-bdd8-40a8-bf20-c009e0c8f366\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qt55f"
Jul 28 01:33:18.808211 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:18.808107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mhln\" (UniqueName: \"kubernetes.io/projected/a083ee12-bdd8-40a8-bf20-c009e0c8f366-kube-api-access-9mhln\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qt55f\" (UID: \"a083ee12-bdd8-40a8-bf20-c009e0c8f366\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qt55f"
Jul 28 01:33:18.808486 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:18.808464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a083ee12-bdd8-40a8-bf20-c009e0c8f366-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qt55f\" (UID: \"a083ee12-bdd8-40a8-bf20-c009e0c8f366\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qt55f"
Jul 28 01:33:18.818576 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:18.818514    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mhln\" (UniqueName: \"kubernetes.io/projected/a083ee12-bdd8-40a8-bf20-c009e0c8f366-kube-api-access-9mhln\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qt55f\" (UID: \"a083ee12-bdd8-40a8-bf20-c009e0c8f366\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qt55f"
Jul 28 01:33:18.877233 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:18.877201    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qt55f"
Jul 28 01:33:19.012363 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:19.012321    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qt55f"]
Jul 28 01:33:19.013975 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:33:19.013945    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda083ee12_bdd8_40a8_bf20_c009e0c8f366.slice/crio-eeb77f32036124606d20e91524135967b8bd116e5c8997b39f96afeb1f1ee571 WatchSource:0}: Error finding container eeb77f32036124606d20e91524135967b8bd116e5c8997b39f96afeb1f1ee571: Status 404 returned error can't find the container with id eeb77f32036124606d20e91524135967b8bd116e5c8997b39f96afeb1f1ee571
Jul 28 01:33:19.593176 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:19.593143    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qt55f" event={"ID":"a083ee12-bdd8-40a8-bf20-c009e0c8f366","Type":"ContainerStarted","Data":"eeb77f32036124606d20e91524135967b8bd116e5c8997b39f96afeb1f1ee571"}
Jul 28 01:33:22.605913 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:22.605872    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qt55f" event={"ID":"a083ee12-bdd8-40a8-bf20-c009e0c8f366","Type":"ContainerStarted","Data":"e77827d531805848aa0e246d8e6ca9f425db283690654d24c18124cbcaa997d5"}
Jul 28 01:33:22.634989 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:22.634931    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qt55f" podStartSLOduration=1.789127813 podStartE2EDuration="4.634914044s" totalImagesPullingTime="2.845786231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:33:22.632988855 +0000 UTC m=+446.210446482" watchObservedRunningTime="2026-07-28 01:33:22.634914044 +0000 UTC m=+446.212371661"
Jul 28 01:33:41.133585 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:41.133553    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m8rzf"]
Jul 28 01:33:41.136619 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:41.136603    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m8rzf"
Jul 28 01:33:41.139674 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:41.139653    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 01:33:41.140897 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:41.140880    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:33:41.140897 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:41.140883    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-rd9h8\""
Jul 28 01:33:41.151221 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:41.151203    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m8rzf"]
Jul 28 01:33:41.284934 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:41.284895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2d8b62d-c846-4e91-8273-25cffbd7040a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-m8rzf\" (UID: \"c2d8b62d-c846-4e91-8273-25cffbd7040a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m8rzf"
Jul 28 01:33:41.285128 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:41.284962    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hklgw\" (UniqueName: \"kubernetes.io/projected/c2d8b62d-c846-4e91-8273-25cffbd7040a-kube-api-access-hklgw\") pod \"openshift-lws-operator-fd8ccff4c-m8rzf\" (UID: \"c2d8b62d-c846-4e91-8273-25cffbd7040a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m8rzf"
Jul 28 01:33:41.386197 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:41.386110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hklgw\" (UniqueName: \"kubernetes.io/projected/c2d8b62d-c846-4e91-8273-25cffbd7040a-kube-api-access-hklgw\") pod \"openshift-lws-operator-fd8ccff4c-m8rzf\" (UID: \"c2d8b62d-c846-4e91-8273-25cffbd7040a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m8rzf"
Jul 28 01:33:41.386197 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:41.386163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2d8b62d-c846-4e91-8273-25cffbd7040a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-m8rzf\" (UID: \"c2d8b62d-c846-4e91-8273-25cffbd7040a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m8rzf"
Jul 28 01:33:41.386523 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:41.386507    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2d8b62d-c846-4e91-8273-25cffbd7040a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-m8rzf\" (UID: \"c2d8b62d-c846-4e91-8273-25cffbd7040a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m8rzf"
Jul 28 01:33:41.394835 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:41.394814    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hklgw\" (UniqueName: \"kubernetes.io/projected/c2d8b62d-c846-4e91-8273-25cffbd7040a-kube-api-access-hklgw\") pod \"openshift-lws-operator-fd8ccff4c-m8rzf\" (UID: \"c2d8b62d-c846-4e91-8273-25cffbd7040a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m8rzf"
Jul 28 01:33:41.445395 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:41.445361    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m8rzf"
Jul 28 01:33:41.606213 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:41.606178    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m8rzf"]
Jul 28 01:33:41.610211 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:33:41.610188    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2d8b62d_c846_4e91_8273_25cffbd7040a.slice/crio-43338a33877514d2991eeb6e38dc8e70ce1c26a6bce088e05b9506037ef92c94 WatchSource:0}: Error finding container 43338a33877514d2991eeb6e38dc8e70ce1c26a6bce088e05b9506037ef92c94: Status 404 returned error can't find the container with id 43338a33877514d2991eeb6e38dc8e70ce1c26a6bce088e05b9506037ef92c94
Jul 28 01:33:41.682305 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:41.682223    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m8rzf" event={"ID":"c2d8b62d-c846-4e91-8273-25cffbd7040a","Type":"ContainerStarted","Data":"43338a33877514d2991eeb6e38dc8e70ce1c26a6bce088e05b9506037ef92c94"}
Jul 28 01:33:44.698624 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:44.698587    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m8rzf" event={"ID":"c2d8b62d-c846-4e91-8273-25cffbd7040a","Type":"ContainerStarted","Data":"60f9d6abc893744e889c2ea46087714e5313188a6a8a08d9f78e0744cffd6725"}
Jul 28 01:33:44.715766 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:33:44.715716    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-m8rzf" podStartSLOduration=1.26193767 podStartE2EDuration="3.715702s" totalImagesPullingTime="2.45376433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:33:44.715193153 +0000 UTC m=+468.292650771" watchObservedRunningTime="2026-07-28 01:33:44.715702 +0000 UTC m=+468.293159618"
Jul 28 01:34:07.153587 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.153557    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"]
Jul 28 01:34:07.159146 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.159123    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.162271 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.162247    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-qvqpw\""
Jul 28 01:34:07.162271 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.162260    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 01:34:07.169939 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.169912    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"]
Jul 28 01:34:07.273377 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.273317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a643be-4bcf-4e39-8705-a5b6e8686d81-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.273565 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.273400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a643be-4bcf-4e39-8705-a5b6e8686d81-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.273565 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.273430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a643be-4bcf-4e39-8705-a5b6e8686d81-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.273565 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.273460    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a643be-4bcf-4e39-8705-a5b6e8686d81-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.273565 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.273483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55a643be-4bcf-4e39-8705-a5b6e8686d81-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.273565 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.273532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a643be-4bcf-4e39-8705-a5b6e8686d81-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.273732 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.273550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbwxp\" (UniqueName: \"kubernetes.io/projected/55a643be-4bcf-4e39-8705-a5b6e8686d81-kube-api-access-jbwxp\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.273732 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.273595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a643be-4bcf-4e39-8705-a5b6e8686d81-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.273732 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.273616    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a643be-4bcf-4e39-8705-a5b6e8686d81-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.374181 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.374145    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a643be-4bcf-4e39-8705-a5b6e8686d81-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.374181 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.374178    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a643be-4bcf-4e39-8705-a5b6e8686d81-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.374449 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.374210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a643be-4bcf-4e39-8705-a5b6e8686d81-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.374449 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.374234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a643be-4bcf-4e39-8705-a5b6e8686d81-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.374449 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.374272    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a643be-4bcf-4e39-8705-a5b6e8686d81-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.374449 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.374309    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a643be-4bcf-4e39-8705-a5b6e8686d81-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.374449 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.374347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55a643be-4bcf-4e39-8705-a5b6e8686d81-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.374449 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.374387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a643be-4bcf-4e39-8705-a5b6e8686d81-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.374449 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.374414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbwxp\" (UniqueName: \"kubernetes.io/projected/55a643be-4bcf-4e39-8705-a5b6e8686d81-kube-api-access-jbwxp\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.374773 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.374598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a643be-4bcf-4e39-8705-a5b6e8686d81-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.374773 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.374715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a643be-4bcf-4e39-8705-a5b6e8686d81-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.374864 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.374827    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a643be-4bcf-4e39-8705-a5b6e8686d81-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.374901 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.374862    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a643be-4bcf-4e39-8705-a5b6e8686d81-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.375066 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.375044    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a643be-4bcf-4e39-8705-a5b6e8686d81-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.376628 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.376605    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a643be-4bcf-4e39-8705-a5b6e8686d81-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.376866 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.376850    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a643be-4bcf-4e39-8705-a5b6e8686d81-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.383475 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.383451    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55a643be-4bcf-4e39-8705-a5b6e8686d81-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.383663 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.383644    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbwxp\" (UniqueName: \"kubernetes.io/projected/55a643be-4bcf-4e39-8705-a5b6e8686d81-kube-api-access-jbwxp\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq\" (UID: \"55a643be-4bcf-4e39-8705-a5b6e8686d81\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.469931 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.469834    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:07.596791 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.596748    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"]
Jul 28 01:34:07.598883 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:34:07.598858    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a643be_4bcf_4e39_8705_a5b6e8686d81.slice/crio-ed613606f6be6d5754a6ea4e7452f39d3c1d67ed4e158be4cefdd34be5cbdb07 WatchSource:0}: Error finding container ed613606f6be6d5754a6ea4e7452f39d3c1d67ed4e158be4cefdd34be5cbdb07: Status 404 returned error can't find the container with id ed613606f6be6d5754a6ea4e7452f39d3c1d67ed4e158be4cefdd34be5cbdb07
Jul 28 01:34:07.788651 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:07.788616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq" event={"ID":"55a643be-4bcf-4e39-8705-a5b6e8686d81","Type":"ContainerStarted","Data":"ed613606f6be6d5754a6ea4e7452f39d3c1d67ed4e158be4cefdd34be5cbdb07"}
Jul 28 01:34:10.910832 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:10.910802    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc"]
Jul 28 01:34:10.915820 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:10.915801    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc"
Jul 28 01:34:10.920094 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:10.920062    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 01:34:10.920223 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:10.920115    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 01:34:10.920300 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:10.920223    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 01:34:10.920720 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:10.920700    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-bw4kj\""
Jul 28 01:34:10.922807 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:10.922787    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc"]
Jul 28 01:34:11.003890 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.003855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f1ea1355-2483-4f56-a5b2-f4967d4199a0-manager-config\") pod \"lws-controller-manager-78db6c7dfd-gspjc\" (UID: \"f1ea1355-2483-4f56-a5b2-f4967d4199a0\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc"
Jul 28 01:34:11.004076 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.003934    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1ea1355-2483-4f56-a5b2-f4967d4199a0-cert\") pod \"lws-controller-manager-78db6c7dfd-gspjc\" (UID: \"f1ea1355-2483-4f56-a5b2-f4967d4199a0\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc"
Jul 28 01:34:11.004076 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.003979    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv8jp\" (UniqueName: \"kubernetes.io/projected/f1ea1355-2483-4f56-a5b2-f4967d4199a0-kube-api-access-bv8jp\") pod \"lws-controller-manager-78db6c7dfd-gspjc\" (UID: \"f1ea1355-2483-4f56-a5b2-f4967d4199a0\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc"
Jul 28 01:34:11.004076 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.004011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1ea1355-2483-4f56-a5b2-f4967d4199a0-metrics-cert\") pod \"lws-controller-manager-78db6c7dfd-gspjc\" (UID: \"f1ea1355-2483-4f56-a5b2-f4967d4199a0\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc"
Jul 28 01:34:11.104914 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.104874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f1ea1355-2483-4f56-a5b2-f4967d4199a0-manager-config\") pod \"lws-controller-manager-78db6c7dfd-gspjc\" (UID: \"f1ea1355-2483-4f56-a5b2-f4967d4199a0\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc"
Jul 28 01:34:11.105099 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.104980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1ea1355-2483-4f56-a5b2-f4967d4199a0-cert\") pod \"lws-controller-manager-78db6c7dfd-gspjc\" (UID: \"f1ea1355-2483-4f56-a5b2-f4967d4199a0\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc"
Jul 28 01:34:11.105099 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.105058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv8jp\" (UniqueName: \"kubernetes.io/projected/f1ea1355-2483-4f56-a5b2-f4967d4199a0-kube-api-access-bv8jp\") pod \"lws-controller-manager-78db6c7dfd-gspjc\" (UID: \"f1ea1355-2483-4f56-a5b2-f4967d4199a0\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc"
Jul 28 01:34:11.105099 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.105091    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1ea1355-2483-4f56-a5b2-f4967d4199a0-metrics-cert\") pod \"lws-controller-manager-78db6c7dfd-gspjc\" (UID: \"f1ea1355-2483-4f56-a5b2-f4967d4199a0\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc"
Jul 28 01:34:11.105676 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.105631    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f1ea1355-2483-4f56-a5b2-f4967d4199a0-manager-config\") pod \"lws-controller-manager-78db6c7dfd-gspjc\" (UID: \"f1ea1355-2483-4f56-a5b2-f4967d4199a0\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc"
Jul 28 01:34:11.107841 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.107816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1ea1355-2483-4f56-a5b2-f4967d4199a0-metrics-cert\") pod \"lws-controller-manager-78db6c7dfd-gspjc\" (UID: \"f1ea1355-2483-4f56-a5b2-f4967d4199a0\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc"
Jul 28 01:34:11.107964 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.107934    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1ea1355-2483-4f56-a5b2-f4967d4199a0-cert\") pod \"lws-controller-manager-78db6c7dfd-gspjc\" (UID: \"f1ea1355-2483-4f56-a5b2-f4967d4199a0\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc"
Jul 28 01:34:11.118599 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.118578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv8jp\" (UniqueName: \"kubernetes.io/projected/f1ea1355-2483-4f56-a5b2-f4967d4199a0-kube-api-access-bv8jp\") pod \"lws-controller-manager-78db6c7dfd-gspjc\" (UID: \"f1ea1355-2483-4f56-a5b2-f4967d4199a0\") " pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc"
Jul 28 01:34:11.228018 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.227934    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc"
Jul 28 01:34:11.336121 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.336018    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 28 01:34:11.336121 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.336109    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 28 01:34:11.336453 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.336150    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 28 01:34:11.426035 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.426005    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc"]
Jul 28 01:34:11.428979 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:34:11.428953    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ea1355_2483_4f56_a5b2_f4967d4199a0.slice/crio-a8e75337e58dad47405098b0ceb6e6b80d1fee7c435ae560667e1513ee38c1af WatchSource:0}: Error finding container a8e75337e58dad47405098b0ceb6e6b80d1fee7c435ae560667e1513ee38c1af: Status 404 returned error can't find the container with id a8e75337e58dad47405098b0ceb6e6b80d1fee7c435ae560667e1513ee38c1af
Jul 28 01:34:11.808177 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.808146    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq" event={"ID":"55a643be-4bcf-4e39-8705-a5b6e8686d81","Type":"ContainerStarted","Data":"5df50f9c9ed4c460ff3bb35fc2a9911cbf21b49faef84717c570bbc4723ddb61"}
Jul 28 01:34:11.809301 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.809277    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc" event={"ID":"f1ea1355-2483-4f56-a5b2-f4967d4199a0","Type":"ContainerStarted","Data":"a8e75337e58dad47405098b0ceb6e6b80d1fee7c435ae560667e1513ee38c1af"}
Jul 28 01:34:11.830573 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:11.830529    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq" podStartSLOduration=1.095626689 podStartE2EDuration="4.830515429s" totalImagesPullingTime="3.73488874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:34:11.829413669 +0000 UTC m=+495.406871285" watchObservedRunningTime="2026-07-28 01:34:11.830515429 +0000 UTC m=+495.407973045"
Jul 28 01:34:12.470658 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:12.470626    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:12.475437 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:12.475416    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:12.813669 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:12.813639    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:12.814708 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:12.814693    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:34:16.830363 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:16.830275    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc" event={"ID":"f1ea1355-2483-4f56-a5b2-f4967d4199a0","Type":"ContainerStarted","Data":"938e8b5f5c572ce90e866583bc2c93920f20c95d61da8fd6632bca8389b1cc4f"}
Jul 28 01:34:16.830702 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:16.830377    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc"
Jul 28 01:34:16.851068 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:16.851030    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc" podStartSLOduration=1.7568413330000001 podStartE2EDuration="6.851017732s" totalImagesPullingTime="5.094176399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:34:16.850563529 +0000 UTC m=+500.428021148" watchObservedRunningTime="2026-07-28 01:34:16.851017732 +0000 UTC m=+500.428475348"
Jul 28 01:34:27.835916 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:27.835885    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-78db6c7dfd-gspjc"
Jul 28 01:34:41.618981 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:41.618947    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-2w4pm"]
Jul 28 01:34:41.622235 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:41.622219    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-2w4pm"
Jul 28 01:34:41.625182 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:41.625161    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 28 01:34:41.625182 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:41.625172    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 01:34:41.625346 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:41.625242    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 28 01:34:41.626482 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:41.626465    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 01:34:41.626569 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:41.626466    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mk4n5\""
Jul 28 01:34:41.631173 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:41.631131    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-2w4pm"]
Jul 28 01:34:41.728577 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:41.728543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2677b484-e75e-438a-82ad-7305228b0b80-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-2w4pm\" (UID: \"2677b484-e75e-438a-82ad-7305228b0b80\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-2w4pm"
Jul 28 01:34:41.728696 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:41.728587    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2677b484-e75e-438a-82ad-7305228b0b80-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-2w4pm\" (UID: \"2677b484-e75e-438a-82ad-7305228b0b80\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-2w4pm"
Jul 28 01:34:41.728696 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:41.728666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sk9v\" (UniqueName: \"kubernetes.io/projected/2677b484-e75e-438a-82ad-7305228b0b80-kube-api-access-7sk9v\") pod \"kuadrant-console-plugin-598f86468b-2w4pm\" (UID: \"2677b484-e75e-438a-82ad-7305228b0b80\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-2w4pm"
Jul 28 01:34:41.829118 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:41.829079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2677b484-e75e-438a-82ad-7305228b0b80-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-2w4pm\" (UID: \"2677b484-e75e-438a-82ad-7305228b0b80\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-2w4pm"
Jul 28 01:34:41.829242 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:41.829131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2677b484-e75e-438a-82ad-7305228b0b80-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-2w4pm\" (UID: \"2677b484-e75e-438a-82ad-7305228b0b80\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-2w4pm"
Jul 28 01:34:41.829242 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:41.829204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sk9v\" (UniqueName: \"kubernetes.io/projected/2677b484-e75e-438a-82ad-7305228b0b80-kube-api-access-7sk9v\") pod \"kuadrant-console-plugin-598f86468b-2w4pm\" (UID: \"2677b484-e75e-438a-82ad-7305228b0b80\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-2w4pm"
Jul 28 01:34:41.829242 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:34:41.829212    2578 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 28 01:34:41.829441 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:34:41.829267    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2677b484-e75e-438a-82ad-7305228b0b80-plugin-serving-cert podName:2677b484-e75e-438a-82ad-7305228b0b80 nodeName:}" failed. No retries permitted until 2026-07-28 01:34:42.329252984 +0000 UTC m=+525.906710582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/2677b484-e75e-438a-82ad-7305228b0b80-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-2w4pm" (UID: "2677b484-e75e-438a-82ad-7305228b0b80") : secret "plugin-serving-cert" not found
Jul 28 01:34:41.829797 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:41.829777    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2677b484-e75e-438a-82ad-7305228b0b80-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-2w4pm\" (UID: \"2677b484-e75e-438a-82ad-7305228b0b80\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-2w4pm"
Jul 28 01:34:41.838219 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:41.838201    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sk9v\" (UniqueName: \"kubernetes.io/projected/2677b484-e75e-438a-82ad-7305228b0b80-kube-api-access-7sk9v\") pod \"kuadrant-console-plugin-598f86468b-2w4pm\" (UID: \"2677b484-e75e-438a-82ad-7305228b0b80\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-2w4pm"
Jul 28 01:34:42.332937 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:42.332905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2677b484-e75e-438a-82ad-7305228b0b80-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-2w4pm\" (UID: \"2677b484-e75e-438a-82ad-7305228b0b80\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-2w4pm"
Jul 28 01:34:42.333092 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:34:42.333040    2578 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 28 01:34:42.333140 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:34:42.333109    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2677b484-e75e-438a-82ad-7305228b0b80-plugin-serving-cert podName:2677b484-e75e-438a-82ad-7305228b0b80 nodeName:}" failed. No retries permitted until 2026-07-28 01:34:43.333091673 +0000 UTC m=+526.910549269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/2677b484-e75e-438a-82ad-7305228b0b80-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-2w4pm" (UID: "2677b484-e75e-438a-82ad-7305228b0b80") : secret "plugin-serving-cert" not found
Jul 28 01:34:43.338559 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:43.338523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2677b484-e75e-438a-82ad-7305228b0b80-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-2w4pm\" (UID: \"2677b484-e75e-438a-82ad-7305228b0b80\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-2w4pm"
Jul 28 01:34:43.340952 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:43.340931    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2677b484-e75e-438a-82ad-7305228b0b80-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-2w4pm\" (UID: \"2677b484-e75e-438a-82ad-7305228b0b80\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-2w4pm"
Jul 28 01:34:43.430536 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:43.430506    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-2w4pm"
Jul 28 01:34:43.568939 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:43.568915    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-2w4pm"]
Jul 28 01:34:43.575864 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:34:43.575841    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2677b484_e75e_438a_82ad_7305228b0b80.slice/crio-9fc605840de3d38b5c4e8616d707cde888c0e06cf4d4f14a1afc276f6630d923 WatchSource:0}: Error finding container 9fc605840de3d38b5c4e8616d707cde888c0e06cf4d4f14a1afc276f6630d923: Status 404 returned error can't find the container with id 9fc605840de3d38b5c4e8616d707cde888c0e06cf4d4f14a1afc276f6630d923
Jul 28 01:34:43.940114 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:43.940086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-2w4pm" event={"ID":"2677b484-e75e-438a-82ad-7305228b0b80","Type":"ContainerStarted","Data":"9fc605840de3d38b5c4e8616d707cde888c0e06cf4d4f14a1afc276f6630d923"}
Jul 28 01:34:48.964628 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:48.964595    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-2w4pm" event={"ID":"2677b484-e75e-438a-82ad-7305228b0b80","Type":"ContainerStarted","Data":"09d125d1c6c8ac50023e816fe7e3dedd6a0bf2ff13cc7559cb8329a7e625f190"}
Jul 28 01:34:48.984989 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:34:48.984947    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-2w4pm" podStartSLOduration=3.156522587 podStartE2EDuration="7.984933383s" totalImagesPullingTime="4.828410796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:34:48.983462079 +0000 UTC m=+532.560919697" watchObservedRunningTime="2026-07-28 01:34:48.984933383 +0000 UTC m=+532.562391018"
Jul 28 01:35:26.601748 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:26.601715    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-v7hsp"]
Jul 28 01:35:26.610624 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:26.610603    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-v7hsp"
Jul 28 01:35:26.618169 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:26.618147    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-v7hsp"]
Jul 28 01:35:26.622264 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:26.622247    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tj9kz\""
Jul 28 01:35:26.676111 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:26.676081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68mq9\" (UniqueName: \"kubernetes.io/projected/d8373c35-6b7a-4c5b-bec0-e3c82490fdde-kube-api-access-68mq9\") pod \"authorino-b54df94b4-v7hsp\" (UID: \"d8373c35-6b7a-4c5b-bec0-e3c82490fdde\") " pod="kuadrant-system/authorino-b54df94b4-v7hsp"
Jul 28 01:35:26.777184 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:26.777153    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68mq9\" (UniqueName: \"kubernetes.io/projected/d8373c35-6b7a-4c5b-bec0-e3c82490fdde-kube-api-access-68mq9\") pod \"authorino-b54df94b4-v7hsp\" (UID: \"d8373c35-6b7a-4c5b-bec0-e3c82490fdde\") " pod="kuadrant-system/authorino-b54df94b4-v7hsp"
Jul 28 01:35:26.791125 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:26.791093    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68mq9\" (UniqueName: \"kubernetes.io/projected/d8373c35-6b7a-4c5b-bec0-e3c82490fdde-kube-api-access-68mq9\") pod \"authorino-b54df94b4-v7hsp\" (UID: \"d8373c35-6b7a-4c5b-bec0-e3c82490fdde\") " pod="kuadrant-system/authorino-b54df94b4-v7hsp"
Jul 28 01:35:26.920106 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:26.920026    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-v7hsp"
Jul 28 01:35:27.072580 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:27.072554    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-v7hsp"]
Jul 28 01:35:27.074627 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:35:27.074598    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8373c35_6b7a_4c5b_bec0_e3c82490fdde.slice/crio-e368604e1cce90e682888de8b245ffe36eb6934c1da7ce7011437654a3878145 WatchSource:0}: Error finding container e368604e1cce90e682888de8b245ffe36eb6934c1da7ce7011437654a3878145: Status 404 returned error can't find the container with id e368604e1cce90e682888de8b245ffe36eb6934c1da7ce7011437654a3878145
Jul 28 01:35:27.108882 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:27.108839    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-v7hsp" event={"ID":"d8373c35-6b7a-4c5b-bec0-e3c82490fdde","Type":"ContainerStarted","Data":"e368604e1cce90e682888de8b245ffe36eb6934c1da7ce7011437654a3878145"}
Jul 28 01:35:29.909970 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:29.909939    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-v7hsp"]
Jul 28 01:35:32.129433 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:32.129401    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-v7hsp" event={"ID":"d8373c35-6b7a-4c5b-bec0-e3c82490fdde","Type":"ContainerStarted","Data":"67fe299ae1cd28e0885698193a7f3a9c46df733a8d1e0b67b90597c0147cf155"}
Jul 28 01:35:32.129941 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:32.129447    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-v7hsp" podUID="d8373c35-6b7a-4c5b-bec0-e3c82490fdde" containerName="authorino" containerID="cri-o://67fe299ae1cd28e0885698193a7f3a9c46df733a8d1e0b67b90597c0147cf155" gracePeriod=30
Jul 28 01:35:32.143402 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:32.143350    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-v7hsp" podStartSLOduration=1.292901292 podStartE2EDuration="6.14331254s" totalImagesPullingTime="4.850411248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:35:32.142749431 +0000 UTC m=+575.720207050" watchObservedRunningTime="2026-07-28 01:35:32.14331254 +0000 UTC m=+575.720770157"
Jul 28 01:35:32.372709 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:32.372686    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-v7hsp"
Jul 28 01:35:32.428247 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:32.428217    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68mq9\" (UniqueName: \"kubernetes.io/projected/d8373c35-6b7a-4c5b-bec0-e3c82490fdde-kube-api-access-68mq9\") pod \"d8373c35-6b7a-4c5b-bec0-e3c82490fdde\" (UID: \"d8373c35-6b7a-4c5b-bec0-e3c82490fdde\") "
Jul 28 01:35:32.430287 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:32.430262    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8373c35-6b7a-4c5b-bec0-e3c82490fdde-kube-api-access-68mq9" (OuterVolumeSpecName: "kube-api-access-68mq9") pod "d8373c35-6b7a-4c5b-bec0-e3c82490fdde" (UID: "d8373c35-6b7a-4c5b-bec0-e3c82490fdde"). InnerVolumeSpecName "kube-api-access-68mq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:35:32.529040 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:32.529010    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68mq9\" (UniqueName: \"kubernetes.io/projected/d8373c35-6b7a-4c5b-bec0-e3c82490fdde-kube-api-access-68mq9\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:35:33.134938 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:33.134896    2578 generic.go:356] "Generic (PLEG): container finished" podID="d8373c35-6b7a-4c5b-bec0-e3c82490fdde" containerID="67fe299ae1cd28e0885698193a7f3a9c46df733a8d1e0b67b90597c0147cf155" exitCode=0
Jul 28 01:35:33.135410 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:33.134951    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-v7hsp"
Jul 28 01:35:33.135410 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:33.134976    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-v7hsp" event={"ID":"d8373c35-6b7a-4c5b-bec0-e3c82490fdde","Type":"ContainerDied","Data":"67fe299ae1cd28e0885698193a7f3a9c46df733a8d1e0b67b90597c0147cf155"}
Jul 28 01:35:33.135410 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:33.135022    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-v7hsp" event={"ID":"d8373c35-6b7a-4c5b-bec0-e3c82490fdde","Type":"ContainerDied","Data":"e368604e1cce90e682888de8b245ffe36eb6934c1da7ce7011437654a3878145"}
Jul 28 01:35:33.135410 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:33.135041    2578 scope.go:117] "RemoveContainer" containerID="67fe299ae1cd28e0885698193a7f3a9c46df733a8d1e0b67b90597c0147cf155"
Jul 28 01:35:33.144883 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:33.144858    2578 scope.go:117] "RemoveContainer" containerID="67fe299ae1cd28e0885698193a7f3a9c46df733a8d1e0b67b90597c0147cf155"
Jul 28 01:35:33.145146 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:35:33.145125    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67fe299ae1cd28e0885698193a7f3a9c46df733a8d1e0b67b90597c0147cf155\": container with ID starting with 67fe299ae1cd28e0885698193a7f3a9c46df733a8d1e0b67b90597c0147cf155 not found: ID does not exist" containerID="67fe299ae1cd28e0885698193a7f3a9c46df733a8d1e0b67b90597c0147cf155"
Jul 28 01:35:33.145209 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:33.145154    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67fe299ae1cd28e0885698193a7f3a9c46df733a8d1e0b67b90597c0147cf155"} err="failed to get container status \"67fe299ae1cd28e0885698193a7f3a9c46df733a8d1e0b67b90597c0147cf155\": rpc error: code = NotFound desc = could not find container \"67fe299ae1cd28e0885698193a7f3a9c46df733a8d1e0b67b90597c0147cf155\": container with ID starting with 67fe299ae1cd28e0885698193a7f3a9c46df733a8d1e0b67b90597c0147cf155 not found: ID does not exist"
Jul 28 01:35:33.151848 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:33.151820    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-v7hsp"]
Jul 28 01:35:33.155220 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:33.155200    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-v7hsp"]
Jul 28 01:35:34.984905 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:34.984869    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8373c35-6b7a-4c5b-bec0-e3c82490fdde" path="/var/lib/kubelet/pods/d8373c35-6b7a-4c5b-bec0-e3c82490fdde/volumes"
Jul 28 01:35:55.762710 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:55.762675    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["observability/jaeger-6bbd7d4c9d-sr4pb"]
Jul 28 01:35:55.763166 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:55.762961    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8373c35-6b7a-4c5b-bec0-e3c82490fdde" containerName="authorino"
Jul 28 01:35:55.763166 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:55.762972    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8373c35-6b7a-4c5b-bec0-e3c82490fdde" containerName="authorino"
Jul 28 01:35:55.763166 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:55.763023    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8373c35-6b7a-4c5b-bec0-e3c82490fdde" containerName="authorino"
Jul 28 01:35:55.765692 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:55.765676    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="observability/jaeger-6bbd7d4c9d-sr4pb"
Jul 28 01:35:55.768749 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:55.768731    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"observability\"/\"jaeger-dockercfg-t4pft\""
Jul 28 01:35:55.769090 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:55.769078    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"observability\"/\"openshift-service-ca.crt\""
Jul 28 01:35:55.770075 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:55.770058    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"observability\"/\"kube-root-ca.crt\""
Jul 28 01:35:55.781127 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:55.781104    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["observability/jaeger-6bbd7d4c9d-sr4pb"]
Jul 28 01:35:55.922255 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:55.922203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x8s5\" (UniqueName: \"kubernetes.io/projected/3b87b086-423e-481e-bb68-11ba9db2d234-kube-api-access-7x8s5\") pod \"jaeger-6bbd7d4c9d-sr4pb\" (UID: \"3b87b086-423e-481e-bb68-11ba9db2d234\") " pod="observability/jaeger-6bbd7d4c9d-sr4pb"
Jul 28 01:35:56.023606 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:56.023524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x8s5\" (UniqueName: \"kubernetes.io/projected/3b87b086-423e-481e-bb68-11ba9db2d234-kube-api-access-7x8s5\") pod \"jaeger-6bbd7d4c9d-sr4pb\" (UID: \"3b87b086-423e-481e-bb68-11ba9db2d234\") " pod="observability/jaeger-6bbd7d4c9d-sr4pb"
Jul 28 01:35:56.032175 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:56.032146    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x8s5\" (UniqueName: \"kubernetes.io/projected/3b87b086-423e-481e-bb68-11ba9db2d234-kube-api-access-7x8s5\") pod \"jaeger-6bbd7d4c9d-sr4pb\" (UID: \"3b87b086-423e-481e-bb68-11ba9db2d234\") " pod="observability/jaeger-6bbd7d4c9d-sr4pb"
Jul 28 01:35:56.075020 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:56.074987    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="observability/jaeger-6bbd7d4c9d-sr4pb"
Jul 28 01:35:56.205522 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:56.205496    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["observability/jaeger-6bbd7d4c9d-sr4pb"]
Jul 28 01:35:56.208150 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:35:56.208125    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b87b086_423e_481e_bb68_11ba9db2d234.slice/crio-4b78840210ef199f1ec2d952755949af4c9f908fc8ff72464a622e22c1f3cde6 WatchSource:0}: Error finding container 4b78840210ef199f1ec2d952755949af4c9f908fc8ff72464a622e22c1f3cde6: Status 404 returned error can't find the container with id 4b78840210ef199f1ec2d952755949af4c9f908fc8ff72464a622e22c1f3cde6
Jul 28 01:35:56.228039 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:56.228009    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="observability/jaeger-6bbd7d4c9d-sr4pb" event={"ID":"3b87b086-423e-481e-bb68-11ba9db2d234","Type":"ContainerStarted","Data":"4b78840210ef199f1ec2d952755949af4c9f908fc8ff72464a622e22c1f3cde6"}
Jul 28 01:35:58.237761 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:58.237716    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="observability/jaeger-6bbd7d4c9d-sr4pb" event={"ID":"3b87b086-423e-481e-bb68-11ba9db2d234","Type":"ContainerStarted","Data":"fabcc8e4e45759134943dbb41698af8e644569c2d866a5ef94e93689f00df3d2"}
Jul 28 01:35:58.238234 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:58.237774    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="observability/jaeger-6bbd7d4c9d-sr4pb"
Jul 28 01:35:58.255342 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:58.255228    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="observability/jaeger-6bbd7d4c9d-sr4pb" podStartSLOduration=1.473408449 podStartE2EDuration="3.25521218s" totalImagesPullingTime="1.781803731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:35:58.25417504 +0000 UTC m=+601.831632656" watchObservedRunningTime="2026-07-28 01:35:58.25521218 +0000 UTC m=+601.832669797"
Jul 28 01:35:59.244251 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:35:59.244223    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="observability/jaeger-6bbd7d4c9d-sr4pb"
Jul 28 01:38:36.881585 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:36.881549    2578 generic.go:356] "Generic (PLEG): container finished" podID="55a643be-4bcf-4e39-8705-a5b6e8686d81" containerID="5df50f9c9ed4c460ff3bb35fc2a9911cbf21b49faef84717c570bbc4723ddb61" exitCode=0
Jul 28 01:38:36.882056 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:36.881624    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq" event={"ID":"55a643be-4bcf-4e39-8705-a5b6e8686d81","Type":"ContainerDied","Data":"5df50f9c9ed4c460ff3bb35fc2a9911cbf21b49faef84717c570bbc4723ddb61"}
Jul 28 01:38:36.882056 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:36.882010    2578 scope.go:117] "RemoveContainer" containerID="5df50f9c9ed4c460ff3bb35fc2a9911cbf21b49faef84717c570bbc4723ddb61"
Jul 28 01:38:36.882905 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:36.882875    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 28 01:38:36.882979 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:36.882960    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 28 01:38:36.883015 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:36.883005    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 28 01:38:37.908681 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:37.908004    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq" event={"ID":"55a643be-4bcf-4e39-8705-a5b6e8686d81","Type":"ContainerStarted","Data":"1a46c85782d22e6264da1bce545e44ceba238c181300c1c05f9dfa4629ae1a92"}
Jul 28 01:38:38.470961 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:38.470923    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:38:38.471945 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:38.471919    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 28 01:38:38.472058 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:38.471969    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq" podUID="55a643be-4bcf-4e39-8705-a5b6e8686d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 28 01:38:39.470780 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:39.470744    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 28 01:38:39.471179 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:39.470812    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq" podUID="55a643be-4bcf-4e39-8705-a5b6e8686d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 28 01:38:40.470542 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:40.470507    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 28 01:38:40.470710 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:40.470573    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq" podUID="55a643be-4bcf-4e39-8705-a5b6e8686d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 28 01:38:41.471119 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:41.471084    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 28 01:38:41.471515 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:41.471154    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq" podUID="55a643be-4bcf-4e39-8705-a5b6e8686d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 28 01:38:42.470645 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:42.470614    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Jul 28 01:38:42.470822 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:42.470671    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq" podUID="55a643be-4bcf-4e39-8705-a5b6e8686d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Jul 28 01:38:42.813663 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:42.813632    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:38:43.475094 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:43.475065    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:38:43.475967 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:43.475949    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq"
Jul 28 01:38:51.537434 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.537400    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"]
Jul 28 01:38:51.539655 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.539638    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.544171 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.544144    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Jul 28 01:38:51.544171 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.544165    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-swjq9\""
Jul 28 01:38:51.544387 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.544221    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Jul 28 01:38:51.544387 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.544266    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Jul 28 01:38:51.553542 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.553520    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"]
Jul 28 01:38:51.555270 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.555248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-7t779\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.555408 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.555278    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0341a4ae-0def-4f8c-88cf-674a3880a281-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-7t779\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.555408 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.555298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-home\") pod \"fc-smoke-test-kserve-7f4687897-7t779\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.555408 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.555321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-dshm\") pod \"fc-smoke-test-kserve-7f4687897-7t779\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.555548 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.555457    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-7t779\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.555548 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.555499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7zjw\" (UniqueName: \"kubernetes.io/projected/0341a4ae-0def-4f8c-88cf-674a3880a281-kube-api-access-m7zjw\") pod \"fc-smoke-test-kserve-7f4687897-7t779\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.655916 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.655886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7zjw\" (UniqueName: \"kubernetes.io/projected/0341a4ae-0def-4f8c-88cf-674a3880a281-kube-api-access-m7zjw\") pod \"fc-smoke-test-kserve-7f4687897-7t779\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.656061 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.655926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-7t779\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.656061 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.655950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0341a4ae-0def-4f8c-88cf-674a3880a281-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-7t779\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.656061 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.655977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-home\") pod \"fc-smoke-test-kserve-7f4687897-7t779\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.656061 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.655998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-dshm\") pod \"fc-smoke-test-kserve-7f4687897-7t779\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.656061 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.656052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-7t779\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.656390 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.656317    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-7t779\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.656450 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.656381    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-home\") pod \"fc-smoke-test-kserve-7f4687897-7t779\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.656490 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.656448    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-7t779\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.658366 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.658325    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-dshm\") pod \"fc-smoke-test-kserve-7f4687897-7t779\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.658640 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.658620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0341a4ae-0def-4f8c-88cf-674a3880a281-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-7t779\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.664861 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.664834    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7zjw\" (UniqueName: \"kubernetes.io/projected/0341a4ae-0def-4f8c-88cf-674a3880a281-kube-api-access-m7zjw\") pod \"fc-smoke-test-kserve-7f4687897-7t779\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.850666 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.850580    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:38:51.981584 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.981524    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"]
Jul 28 01:38:51.983790 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:38:51.983762    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0341a4ae_0def_4f8c_88cf_674a3880a281.slice/crio-bcdc337d0924fc618c22f7632367677e0475998eee7cd0f7ec4e262be3183d04 WatchSource:0}: Error finding container bcdc337d0924fc618c22f7632367677e0475998eee7cd0f7ec4e262be3183d04: Status 404 returned error can't find the container with id bcdc337d0924fc618c22f7632367677e0475998eee7cd0f7ec4e262be3183d04
Jul 28 01:38:51.985752 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:51.985734    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:38:52.974205 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:52.974167    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779" event={"ID":"0341a4ae-0def-4f8c-88cf-674a3880a281","Type":"ContainerStarted","Data":"bcdc337d0924fc618c22f7632367677e0475998eee7cd0f7ec4e262be3183d04"}
Jul 28 01:38:53.979614 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:53.979558    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779" event={"ID":"0341a4ae-0def-4f8c-88cf-674a3880a281","Type":"ContainerStarted","Data":"6f33b83dc7795c65456c6c140c59bc713fc40265d99d3cb8380ef2ab7f3689d3"}
Jul 28 01:38:53.996552 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:53.996508    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779" podStartSLOduration=1.484974226 podStartE2EDuration="2.996492333s" totalImagesPullingTime="1.511518107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:38:53.995889157 +0000 UTC m=+777.573346807" watchObservedRunningTime="2026-07-28 01:38:53.996492333 +0000 UTC m=+777.573949950"
Jul 28 01:38:54.851146 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:38:54.851108    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:39:01.851752 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:01.851713    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:39:01.864453 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:01.864424    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:39:02.025911 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:02.025875    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:39:33.712830 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:33.712750    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"]
Jul 28 01:39:33.721380 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:39:33.721346    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:39:33.721380 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:39:33.721380 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:39:33.721380 ip-10-0-139-81 kubenswrapper[2578]:  > containerID="6f33b83dc7795c65456c6c140c59bc713fc40265d99d3cb8380ef2ab7f3689d3" cmd=["/bin/sleep","15"]
Jul 28 01:39:33.721632 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:33.721398    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779" message="" err=<
Jul 28 01:39:33.721632 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:39:33.721632 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:39:33.721632 ip-10-0-139-81 kubenswrapper[2578]:  >
Jul 28 01:39:33.721632 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:39:33.721410    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:39:33.721632 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:39:33.721632 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:39:33.721632 ip-10-0-139-81 kubenswrapper[2578]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779" podUID="0341a4ae-0def-4f8c-88cf-674a3880a281" containerName="main" containerID="cri-o://6f33b83dc7795c65456c6c140c59bc713fc40265d99d3cb8380ef2ab7f3689d3"
Jul 28 01:39:33.721632 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:33.721452    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779" podUID="0341a4ae-0def-4f8c-88cf-674a3880a281" containerName="main" containerID="cri-o://6f33b83dc7795c65456c6c140c59bc713fc40265d99d3cb8380ef2ab7f3689d3" gracePeriod=60
Jul 28 01:39:34.063080 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.063058    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:39:34.152304 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.152274    2578 generic.go:356] "Generic (PLEG): container finished" podID="0341a4ae-0def-4f8c-88cf-674a3880a281" containerID="6f33b83dc7795c65456c6c140c59bc713fc40265d99d3cb8380ef2ab7f3689d3" exitCode=0
Jul 28 01:39:34.152474 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.152315    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"
Jul 28 01:39:34.152474 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.152369    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779" event={"ID":"0341a4ae-0def-4f8c-88cf-674a3880a281","Type":"ContainerDied","Data":"6f33b83dc7795c65456c6c140c59bc713fc40265d99d3cb8380ef2ab7f3689d3"}
Jul 28 01:39:34.152474 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.152430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779" event={"ID":"0341a4ae-0def-4f8c-88cf-674a3880a281","Type":"ContainerDied","Data":"bcdc337d0924fc618c22f7632367677e0475998eee7cd0f7ec4e262be3183d04"}
Jul 28 01:39:34.152474 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.152459    2578 scope.go:117] "RemoveContainer" containerID="6f33b83dc7795c65456c6c140c59bc713fc40265d99d3cb8380ef2ab7f3689d3"
Jul 28 01:39:34.160144 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.160131    2578 scope.go:117] "RemoveContainer" containerID="6f33b83dc7795c65456c6c140c59bc713fc40265d99d3cb8380ef2ab7f3689d3"
Jul 28 01:39:34.160432 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:39:34.160415    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f33b83dc7795c65456c6c140c59bc713fc40265d99d3cb8380ef2ab7f3689d3\": container with ID starting with 6f33b83dc7795c65456c6c140c59bc713fc40265d99d3cb8380ef2ab7f3689d3 not found: ID does not exist" containerID="6f33b83dc7795c65456c6c140c59bc713fc40265d99d3cb8380ef2ab7f3689d3"
Jul 28 01:39:34.160486 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.160440    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f33b83dc7795c65456c6c140c59bc713fc40265d99d3cb8380ef2ab7f3689d3"} err="failed to get container status \"6f33b83dc7795c65456c6c140c59bc713fc40265d99d3cb8380ef2ab7f3689d3\": rpc error: code = NotFound desc = could not find container \"6f33b83dc7795c65456c6c140c59bc713fc40265d99d3cb8380ef2ab7f3689d3\": container with ID starting with 6f33b83dc7795c65456c6c140c59bc713fc40265d99d3cb8380ef2ab7f3689d3 not found: ID does not exist"
Jul 28 01:39:34.187713 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.187695    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-model-cache\") pod \"0341a4ae-0def-4f8c-88cf-674a3880a281\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") "
Jul 28 01:39:34.187823 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.187740    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-home\") pod \"0341a4ae-0def-4f8c-88cf-674a3880a281\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") "
Jul 28 01:39:34.187823 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.187774    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-tmp-dir\") pod \"0341a4ae-0def-4f8c-88cf-674a3880a281\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") "
Jul 28 01:39:34.187929 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.187839    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0341a4ae-0def-4f8c-88cf-674a3880a281-tls-certs\") pod \"0341a4ae-0def-4f8c-88cf-674a3880a281\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") "
Jul 28 01:39:34.187929 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.187892    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-dshm\") pod \"0341a4ae-0def-4f8c-88cf-674a3880a281\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") "
Jul 28 01:39:34.188029 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.187928    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7zjw\" (UniqueName: \"kubernetes.io/projected/0341a4ae-0def-4f8c-88cf-674a3880a281-kube-api-access-m7zjw\") pod \"0341a4ae-0def-4f8c-88cf-674a3880a281\" (UID: \"0341a4ae-0def-4f8c-88cf-674a3880a281\") "
Jul 28 01:39:34.188029 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.187935    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-model-cache" (OuterVolumeSpecName: "model-cache") pod "0341a4ae-0def-4f8c-88cf-674a3880a281" (UID: "0341a4ae-0def-4f8c-88cf-674a3880a281"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:39:34.188029 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.188016    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-home" (OuterVolumeSpecName: "home") pod "0341a4ae-0def-4f8c-88cf-674a3880a281" (UID: "0341a4ae-0def-4f8c-88cf-674a3880a281"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:39:34.188176 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.188051    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0341a4ae-0def-4f8c-88cf-674a3880a281" (UID: "0341a4ae-0def-4f8c-88cf-674a3880a281"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:39:34.188176 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.188163    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:39:34.188283 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.188180    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:39:34.188283 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.188193    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:39:34.190004 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.189978    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-dshm" (OuterVolumeSpecName: "dshm") pod "0341a4ae-0def-4f8c-88cf-674a3880a281" (UID: "0341a4ae-0def-4f8c-88cf-674a3880a281"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:39:34.190086 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.189980    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0341a4ae-0def-4f8c-88cf-674a3880a281-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0341a4ae-0def-4f8c-88cf-674a3880a281" (UID: "0341a4ae-0def-4f8c-88cf-674a3880a281"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:39:34.190086 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.190006    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0341a4ae-0def-4f8c-88cf-674a3880a281-kube-api-access-m7zjw" (OuterVolumeSpecName: "kube-api-access-m7zjw") pod "0341a4ae-0def-4f8c-88cf-674a3880a281" (UID: "0341a4ae-0def-4f8c-88cf-674a3880a281"). InnerVolumeSpecName "kube-api-access-m7zjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:39:34.288570 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.288530    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0341a4ae-0def-4f8c-88cf-674a3880a281-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:39:34.288570 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.288556    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0341a4ae-0def-4f8c-88cf-674a3880a281-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:39:34.288570 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.288566    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7zjw\" (UniqueName: \"kubernetes.io/projected/0341a4ae-0def-4f8c-88cf-674a3880a281-kube-api-access-m7zjw\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:39:34.475753 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.475724    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"]
Jul 28 01:39:34.479843 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.479820    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779"]
Jul 28 01:39:34.987525 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:34.986051    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0341a4ae-0def-4f8c-88cf-674a3880a281" path="/var/lib/kubelet/pods/0341a4ae-0def-4f8c-88cf-674a3880a281/volumes"
Jul 28 01:39:35.015841 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:39:35.015792    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-7t779" podUID="0341a4ae-0def-4f8c-88cf-674a3880a281" containerName="main" probeResult="failure" output="Get \"https://10.134.0.25:8000/health\": context deadline exceeded"
Jul 28 01:40:55.861491 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:55.861293    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"]
Jul 28 01:40:55.861913 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:55.861608    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0341a4ae-0def-4f8c-88cf-674a3880a281" containerName="main"
Jul 28 01:40:55.861913 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:55.861623    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0341a4ae-0def-4f8c-88cf-674a3880a281" containerName="main"
Jul 28 01:40:55.861913 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:55.861687    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0341a4ae-0def-4f8c-88cf-674a3880a281" containerName="main"
Jul 28 01:40:55.864750 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:55.864732    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:55.872325 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:55.872305    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Jul 28 01:40:55.872444 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:55.872382    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-khnnd\""
Jul 28 01:40:55.872444 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:55.872384    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Jul 28 01:40:55.872648 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:55.872633    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Jul 28 01:40:55.884810 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:55.884786    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"]
Jul 28 01:40:56.032857 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.032803    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.032857 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.032856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.033146 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.032887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.033146 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.032959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.033146 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.033037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.033146 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.033079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.033146 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.033103    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzxsn\" (UniqueName: \"kubernetes.io/projected/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-kube-api-access-xzxsn\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.033146 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.033134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.033406 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.033158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.134024 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.133912    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.134024 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.133980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.134267 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.134039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.134267 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.134075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.134267 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.134096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzxsn\" (UniqueName: \"kubernetes.io/projected/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-kube-api-access-xzxsn\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.134267 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.134121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.134267 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.134141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.134267 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.134170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.134267 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.134191    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.134714 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.134532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.134714 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.134629    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.134813 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.134741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.134907 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.134883    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.134971 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.134955    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.136468 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.136445    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.136657 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.136636    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.143070 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.143033    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzxsn\" (UniqueName: \"kubernetes.io/projected/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-kube-api-access-xzxsn\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.143170 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.143151    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-bbtgt\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.175938 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.175904    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:56.310497 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.310473    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"]
Jul 28 01:40:56.312919 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:40:56.312891    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4faf0ee_a6c4_4b68_b6bb_08c18d6f72f9.slice/crio-29745053393b1a87073766f15d3dcc137ea566b3de6f93f5d37e37fda88efa43 WatchSource:0}: Error finding container 29745053393b1a87073766f15d3dcc137ea566b3de6f93f5d37e37fda88efa43: Status 404 returned error can't find the container with id 29745053393b1a87073766f15d3dcc137ea566b3de6f93f5d37e37fda88efa43
Jul 28 01:40:56.314915 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.314886    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 28 01:40:56.314979 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.314950    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 28 01:40:56.315025 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.315013    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 28 01:40:56.485161 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.485072    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt" event={"ID":"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9","Type":"ContainerStarted","Data":"ae52aad1ce54ee742fa1191c02659ba059f7f2e27beb48529f262c01b7c19fe0"}
Jul 28 01:40:56.485309 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.485167    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt" event={"ID":"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9","Type":"ContainerStarted","Data":"29745053393b1a87073766f15d3dcc137ea566b3de6f93f5d37e37fda88efa43"}
Jul 28 01:40:56.506264 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:56.506212    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt" podStartSLOduration=1.506191402 podStartE2EDuration="1.506191402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:40:56.505369204 +0000 UTC m=+900.082826850" watchObservedRunningTime="2026-07-28 01:40:56.506191402 +0000 UTC m=+900.083649020"
Jul 28 01:40:57.176931 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:57.176898    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:57.181923 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:57.181897    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:57.489371 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:57.489256    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:40:57.490320 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:40:57.490300    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:41:05.572099 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.572067    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"]
Jul 28 01:41:05.575472 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.575456    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.579940 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.579918    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Jul 28 01:41:05.580047 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.579938    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-xp6x9\""
Jul 28 01:41:05.580402 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.580386    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Jul 28 01:41:05.581087 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.581010    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Jul 28 01:41:05.596102 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.595200    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"]
Jul 28 01:41:05.621066 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.621039    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.621255 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.621242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52eec79c-d8ae-4d51-aa86-818b4efc1905-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.621387 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.621374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbncn\" (UniqueName: \"kubernetes.io/projected/52eec79c-d8ae-4d51-aa86-818b4efc1905-kube-api-access-cbncn\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.621498 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.621487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.621579 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.621570    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.621660 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.621652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.621738 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.621730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.621827 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.621816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.621925 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.621916    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.722375 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.722311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.722598 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.722584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52eec79c-d8ae-4d51-aa86-818b4efc1905-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.722697 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.722676    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.722807 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.722795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbncn\" (UniqueName: \"kubernetes.io/projected/52eec79c-d8ae-4d51-aa86-818b4efc1905-kube-api-access-cbncn\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.722923 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.722903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.723014 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.722938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.723141 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.723126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.723259 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.723238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52eec79c-d8ae-4d51-aa86-818b4efc1905-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.723419 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.723256    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.723490 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.723395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.723490 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.723282    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.723606 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.723489    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.723770 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.723751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.723770 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.723762    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.725119 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.725098    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.725682 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.725666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.732257 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.732091    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.732257 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.732176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbncn\" (UniqueName: \"kubernetes.io/projected/52eec79c-d8ae-4d51-aa86-818b4efc1905-kube-api-access-cbncn\") pod \"router-gateway-1-openshift-default-66bbccc5fb-5wn88\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:05.886280 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:05.886182    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:06.063230 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:06.063205    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"]
Jul 28 01:41:06.065342 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:41:06.065305    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52eec79c_d8ae_4d51_aa86_818b4efc1905.slice/crio-d5349dc7562185779d067b296e10989624bd2d09ecdedff9645638686059536f WatchSource:0}: Error finding container d5349dc7562185779d067b296e10989624bd2d09ecdedff9645638686059536f: Status 404 returned error can't find the container with id d5349dc7562185779d067b296e10989624bd2d09ecdedff9645638686059536f
Jul 28 01:41:06.067536 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:06.067506    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 28 01:41:06.067652 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:06.067614    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 28 01:41:06.067714 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:06.067660    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 28 01:41:06.535606 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:06.535570    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88" event={"ID":"52eec79c-d8ae-4d51-aa86-818b4efc1905","Type":"ContainerStarted","Data":"16b082d4ddf72a058793e26851c159bc3cb88aad2e24e539529f5ad7e4d81101"}
Jul 28 01:41:06.535789 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:06.535624    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88" event={"ID":"52eec79c-d8ae-4d51-aa86-818b4efc1905","Type":"ContainerStarted","Data":"d5349dc7562185779d067b296e10989624bd2d09ecdedff9645638686059536f"}
Jul 28 01:41:06.559921 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:06.559863    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88" podStartSLOduration=1.559846788 podStartE2EDuration="1.559846788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:41:06.558893154 +0000 UTC m=+910.136350776" watchObservedRunningTime="2026-07-28 01:41:06.559846788 +0000 UTC m=+910.137304459"
Jul 28 01:41:06.886768 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:06.886676    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:07.900956 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:07.900920    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:08.544609 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:08.544581    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:08.545564 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:08.545542    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:10.240065 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:10.240028    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"]
Jul 28 01:41:10.240459 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:10.240241    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt" podUID="f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9" containerName="istio-proxy" containerID="cri-o://ae52aad1ce54ee742fa1191c02659ba059f7f2e27beb48529f262c01b7c19fe0" gracePeriod=30
Jul 28 01:41:12.908781 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:12.907846    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt" podUID="f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": read tcp 10.134.0.2:36278->10.134.0.26:15021: read: connection reset by peer"
Jul 28 01:41:13.108547 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.108525    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:41:13.185817 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.185781    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-podinfo\") pod \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") "
Jul 28 01:41:13.185992 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.185829    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-token\") pod \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") "
Jul 28 01:41:13.185992 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.185863    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-workload-certs\") pod \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") "
Jul 28 01:41:13.185992 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.185897    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istiod-ca-cert\") pod \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") "
Jul 28 01:41:13.185992 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.185922    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-envoy\") pod \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") "
Jul 28 01:41:13.185992 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.185968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-credential-socket\") pod \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") "
Jul 28 01:41:13.187132 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.185995    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzxsn\" (UniqueName: \"kubernetes.io/projected/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-kube-api-access-xzxsn\") pod \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") "
Jul 28 01:41:13.187132 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.186184    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9" (UID: "f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:13.187132 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.186354    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9" (UID: "f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:41:13.187132 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.186362    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9" (UID: "f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:13.187132 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.186488    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-workload-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:41:13.187132 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.186508    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istiod-ca-cert\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:41:13.187132 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.186522    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-credential-socket\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:41:13.188439 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.188398    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9" (UID: "f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 01:41:13.188439 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.188425    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-token" (OuterVolumeSpecName: "istio-token") pod "f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9" (UID: "f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:41:13.188600 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.188399    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9" (UID: "f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:13.188600 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.188512    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-kube-api-access-xzxsn" (OuterVolumeSpecName: "kube-api-access-xzxsn") pod "f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9" (UID: "f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9"). InnerVolumeSpecName "kube-api-access-xzxsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:41:13.286950 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.286914    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-workload-socket\") pod \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") "
Jul 28 01:41:13.286950 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.286950    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-data\") pod \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\" (UID: \"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9\") "
Jul 28 01:41:13.287197 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.287102    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-envoy\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:41:13.287197 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.287113    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzxsn\" (UniqueName: \"kubernetes.io/projected/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-kube-api-access-xzxsn\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:41:13.287197 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.287122    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-podinfo\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:41:13.287197 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.287132    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-token\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:41:13.287385 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.287216    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9" (UID: "f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:13.287385 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.287222    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-data" (OuterVolumeSpecName: "istio-data") pod "f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9" (UID: "f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:13.387958 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.387931    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-workload-socket\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:41:13.387958 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.387955    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9-istio-data\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:41:13.573651 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.573624    2578 generic.go:356] "Generic (PLEG): container finished" podID="f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9" containerID="ae52aad1ce54ee742fa1191c02659ba059f7f2e27beb48529f262c01b7c19fe0" exitCode=0
Jul 28 01:41:13.573817 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.573670    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"
Jul 28 01:41:13.573817 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.573672    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt" event={"ID":"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9","Type":"ContainerDied","Data":"ae52aad1ce54ee742fa1191c02659ba059f7f2e27beb48529f262c01b7c19fe0"}
Jul 28 01:41:13.573817 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.573793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt" event={"ID":"f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9","Type":"ContainerDied","Data":"29745053393b1a87073766f15d3dcc137ea566b3de6f93f5d37e37fda88efa43"}
Jul 28 01:41:13.573817 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.573817    2578 scope.go:117] "RemoveContainer" containerID="ae52aad1ce54ee742fa1191c02659ba059f7f2e27beb48529f262c01b7c19fe0"
Jul 28 01:41:13.582381 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.582354    2578 scope.go:117] "RemoveContainer" containerID="ae52aad1ce54ee742fa1191c02659ba059f7f2e27beb48529f262c01b7c19fe0"
Jul 28 01:41:13.582645 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:41:13.582626    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae52aad1ce54ee742fa1191c02659ba059f7f2e27beb48529f262c01b7c19fe0\": container with ID starting with ae52aad1ce54ee742fa1191c02659ba059f7f2e27beb48529f262c01b7c19fe0 not found: ID does not exist" containerID="ae52aad1ce54ee742fa1191c02659ba059f7f2e27beb48529f262c01b7c19fe0"
Jul 28 01:41:13.582698 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.582654    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae52aad1ce54ee742fa1191c02659ba059f7f2e27beb48529f262c01b7c19fe0"} err="failed to get container status \"ae52aad1ce54ee742fa1191c02659ba059f7f2e27beb48529f262c01b7c19fe0\": rpc error: code = NotFound desc = could not find container \"ae52aad1ce54ee742fa1191c02659ba059f7f2e27beb48529f262c01b7c19fe0\": container with ID starting with ae52aad1ce54ee742fa1191c02659ba059f7f2e27beb48529f262c01b7c19fe0 not found: ID does not exist"
Jul 28 01:41:13.594436 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.594415    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"]
Jul 28 01:41:13.599568 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:13.599549    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-bbtgt"]
Jul 28 01:41:14.986390 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:14.986356    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9" path="/var/lib/kubelet/pods/f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9/volumes"
Jul 28 01:41:25.233860 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:25.233771    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"]
Jul 28 01:41:25.234243 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:25.233978    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88" podUID="52eec79c-d8ae-4d51-aa86-818b4efc1905" containerName="istio-proxy" containerID="cri-o://16b082d4ddf72a058793e26851c159bc3cb88aad2e24e539529f5ad7e4d81101" gracePeriod=30
Jul 28 01:41:27.646031 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.645997    2578 generic.go:356] "Generic (PLEG): container finished" podID="52eec79c-d8ae-4d51-aa86-818b4efc1905" containerID="16b082d4ddf72a058793e26851c159bc3cb88aad2e24e539529f5ad7e4d81101" exitCode=0
Jul 28 01:41:27.646452 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.646071    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88" event={"ID":"52eec79c-d8ae-4d51-aa86-818b4efc1905","Type":"ContainerDied","Data":"16b082d4ddf72a058793e26851c159bc3cb88aad2e24e539529f5ad7e4d81101"}
Jul 28 01:41:27.675886 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.675863    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:27.700647 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.700620    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52eec79c-d8ae-4d51-aa86-818b4efc1905-istiod-ca-cert\") pod \"52eec79c-d8ae-4d51-aa86-818b4efc1905\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") "
Jul 28 01:41:27.700790 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.700654    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-credential-socket\") pod \"52eec79c-d8ae-4d51-aa86-818b4efc1905\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") "
Jul 28 01:41:27.700790 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.700695    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-token\") pod \"52eec79c-d8ae-4d51-aa86-818b4efc1905\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") "
Jul 28 01:41:27.700790 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.700716    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-podinfo\") pod \"52eec79c-d8ae-4d51-aa86-818b4efc1905\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") "
Jul 28 01:41:27.700790 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.700733    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-workload-socket\") pod \"52eec79c-d8ae-4d51-aa86-818b4efc1905\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") "
Jul 28 01:41:27.700790 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.700750    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-workload-certs\") pod \"52eec79c-d8ae-4d51-aa86-818b4efc1905\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") "
Jul 28 01:41:27.700790 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.700784    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-envoy\") pod \"52eec79c-d8ae-4d51-aa86-818b4efc1905\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") "
Jul 28 01:41:27.701087 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.700809    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-data\") pod \"52eec79c-d8ae-4d51-aa86-818b4efc1905\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") "
Jul 28 01:41:27.701087 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.700838    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbncn\" (UniqueName: \"kubernetes.io/projected/52eec79c-d8ae-4d51-aa86-818b4efc1905-kube-api-access-cbncn\") pod \"52eec79c-d8ae-4d51-aa86-818b4efc1905\" (UID: \"52eec79c-d8ae-4d51-aa86-818b4efc1905\") "
Jul 28 01:41:27.701087 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.701025    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52eec79c-d8ae-4d51-aa86-818b4efc1905-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "52eec79c-d8ae-4d51-aa86-818b4efc1905" (UID: "52eec79c-d8ae-4d51-aa86-818b4efc1905"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:41:27.701235 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.701215    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "52eec79c-d8ae-4d51-aa86-818b4efc1905" (UID: "52eec79c-d8ae-4d51-aa86-818b4efc1905"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:27.701490 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.701415    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "52eec79c-d8ae-4d51-aa86-818b4efc1905" (UID: "52eec79c-d8ae-4d51-aa86-818b4efc1905"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:27.701490 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.701478    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-data" (OuterVolumeSpecName: "istio-data") pod "52eec79c-d8ae-4d51-aa86-818b4efc1905" (UID: "52eec79c-d8ae-4d51-aa86-818b4efc1905"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:27.702679 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.702639    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "52eec79c-d8ae-4d51-aa86-818b4efc1905" (UID: "52eec79c-d8ae-4d51-aa86-818b4efc1905"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:27.706035 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.704883    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-token" (OuterVolumeSpecName: "istio-token") pod "52eec79c-d8ae-4d51-aa86-818b4efc1905" (UID: "52eec79c-d8ae-4d51-aa86-818b4efc1905"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:41:27.706035 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.705130    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "52eec79c-d8ae-4d51-aa86-818b4efc1905" (UID: "52eec79c-d8ae-4d51-aa86-818b4efc1905"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 01:41:27.706035 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.705132    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "52eec79c-d8ae-4d51-aa86-818b4efc1905" (UID: "52eec79c-d8ae-4d51-aa86-818b4efc1905"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:41:27.707097 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.707064    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52eec79c-d8ae-4d51-aa86-818b4efc1905-kube-api-access-cbncn" (OuterVolumeSpecName: "kube-api-access-cbncn") pod "52eec79c-d8ae-4d51-aa86-818b4efc1905" (UID: "52eec79c-d8ae-4d51-aa86-818b4efc1905"). InnerVolumeSpecName "kube-api-access-cbncn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:41:27.803825 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.803789    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-token\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:41:27.803825 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.803828    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-podinfo\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:41:27.803998 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.803842    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-workload-socket\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:41:27.803998 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.803853    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-workload-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:41:27.803998 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.803866    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-envoy\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:41:27.803998 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.803877    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-istio-data\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:41:27.803998 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.803887    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbncn\" (UniqueName: \"kubernetes.io/projected/52eec79c-d8ae-4d51-aa86-818b4efc1905-kube-api-access-cbncn\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:41:27.803998 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.803896    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52eec79c-d8ae-4d51-aa86-818b4efc1905-istiod-ca-cert\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:41:27.803998 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:27.803904    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52eec79c-d8ae-4d51-aa86-818b4efc1905-credential-socket\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:41:28.656545 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:28.656502    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88" event={"ID":"52eec79c-d8ae-4d51-aa86-818b4efc1905","Type":"ContainerDied","Data":"d5349dc7562185779d067b296e10989624bd2d09ecdedff9645638686059536f"}
Jul 28 01:41:28.656545 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:28.656548    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"
Jul 28 01:41:28.657054 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:28.656608    2578 scope.go:117] "RemoveContainer" containerID="16b082d4ddf72a058793e26851c159bc3cb88aad2e24e539529f5ad7e4d81101"
Jul 28 01:41:28.677913 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:28.677870    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"]
Jul 28 01:41:28.681369 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:28.681323    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-5wn88"]
Jul 28 01:41:28.984480 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:28.984406    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52eec79c-d8ae-4d51-aa86-818b4efc1905" path="/var/lib/kubelet/pods/52eec79c-d8ae-4d51-aa86-818b4efc1905/volumes"
Jul 28 01:41:56.292167 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.292123    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 01:41:56.292697 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.292571    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9" containerName="istio-proxy"
Jul 28 01:41:56.292697 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.292590    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9" containerName="istio-proxy"
Jul 28 01:41:56.292697 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.292607    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52eec79c-d8ae-4d51-aa86-818b4efc1905" containerName="istio-proxy"
Jul 28 01:41:56.292697 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.292616    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="52eec79c-d8ae-4d51-aa86-818b4efc1905" containerName="istio-proxy"
Jul 28 01:41:56.292697 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.292684    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="52eec79c-d8ae-4d51-aa86-818b4efc1905" containerName="istio-proxy"
Jul 28 01:41:56.292697 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.292697    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4faf0ee-a6c4-4b68-b6bb-08c18d6f72f9" containerName="istio-proxy"
Jul 28 01:41:56.295157 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.295132    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.297890 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.297862    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-2pbh9\""
Jul 28 01:41:56.298097 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.298045    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 28 01:41:56.298475 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.298448    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-6sntk\""
Jul 28 01:41:56.299289 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.299269    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 28 01:41:56.299489 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.299461    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 28 01:41:56.299688 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.299480    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 28 01:41:56.315085 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.315057    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 01:41:56.436682 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.436640    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.436911 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.436703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.436911 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.436729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.436911 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.436790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.436911 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.436810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.436911 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.436847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.436911 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.436863    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x72qg\" (UniqueName: \"kubernetes.io/projected/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-kube-api-access-x72qg\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.538014 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.537976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.538014 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.538022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.538250 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.538048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.538250 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.538085    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.538250 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.538103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.538250 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.538149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.538250 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.538177    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x72qg\" (UniqueName: \"kubernetes.io/projected/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-kube-api-access-x72qg\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.538559 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.538518    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.538693 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.538589    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.538693 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.538631    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.538693 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.538656    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.540467 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.540440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.540568 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.540551    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.550779 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.550724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x72qg\" (UniqueName: \"kubernetes.io/projected/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-kube-api-access-x72qg\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.614912 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.614875    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:41:56.761703 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.761672    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 01:41:56.764259 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:41:56.764235    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab8e1ce4_ec42_4b66_bdcd_2e2976fb066a.slice/crio-5c54882c4572783990b9c33f1338fc0d3aace01b5d2753be8c8b4f9f6ee5b55f WatchSource:0}: Error finding container 5c54882c4572783990b9c33f1338fc0d3aace01b5d2753be8c8b4f9f6ee5b55f: Status 404 returned error can't find the container with id 5c54882c4572783990b9c33f1338fc0d3aace01b5d2753be8c8b4f9f6ee5b55f
Jul 28 01:41:56.775467 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:41:56.775440    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a","Type":"ContainerStarted","Data":"5c54882c4572783990b9c33f1338fc0d3aace01b5d2753be8c8b4f9f6ee5b55f"}
Jul 28 01:42:00.551463 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:42:00.551433    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 28 01:42:00.799252 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:42:00.799141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a","Type":"ContainerStarted","Data":"15fe15daa2aaed556038a58a9aea191b3babe19c35d250bd38fe464d64df2133"}
Jul 28 01:42:05.824563 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:42:05.824527    2578 generic.go:356] "Generic (PLEG): container finished" podID="ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" containerID="15fe15daa2aaed556038a58a9aea191b3babe19c35d250bd38fe464d64df2133" exitCode=0
Jul 28 01:42:05.824563 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:42:05.824569    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a","Type":"ContainerDied","Data":"15fe15daa2aaed556038a58a9aea191b3babe19c35d250bd38fe464d64df2133"}
Jul 28 01:42:36.992673 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:42:36.992637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a","Type":"ContainerStarted","Data":"daea4507c419d5084592fd75908e19da62a9a40ca9c466dbd68ff1b53fa41701"}
Jul 28 01:42:37.013229 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:42:37.013179    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.479559544 podStartE2EDuration="41.013164746s" totalImagesPullingTime="33.926696665s" totalInitContainerRuntime="4.606908537s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:41:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:42:37.011857445 +0000 UTC m=+1000.589315072" watchObservedRunningTime="2026-07-28 01:42:37.013164746 +0000 UTC m=+1000.590622363"
Jul 28 01:42:37.615235 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:42:37.615198    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:42:46.615701 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:42:46.615663    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:42:46.616983 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:42:46.616952    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 01:42:56.615511 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:42:56.615471    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 01:43:06.615867 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:43:06.615824    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 01:43:16.615754 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:43:16.615698    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 01:43:26.615973 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:43:26.615924    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 01:43:36.616118 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:43:36.616067    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 01:43:46.615257 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:43:46.615217    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 01:43:56.615693 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:43:56.615602    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 01:44:06.616052 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:06.616004    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": dial tcp 10.134.0.28:8000: connect: connection refused"
Jul 28 01:44:16.624903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:16.624872    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:44:16.632726 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:16.632698    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:44:38.636268 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:38.636233    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 01:44:53.658453 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:53.658400    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" containerName="main" containerID="cri-o://daea4507c419d5084592fd75908e19da62a9a40ca9c466dbd68ff1b53fa41701" gracePeriod=45
Jul 28 01:44:54.408917 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.408893    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:44:54.519975 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.519948    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-tmp-dir\") pod \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") "
Jul 28 01:44:54.520147 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.519989    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-dshm\") pod \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") "
Jul 28 01:44:54.520147 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.520013    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-model-cache\") pod \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") "
Jul 28 01:44:54.520147 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.520043    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-home\") pod \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") "
Jul 28 01:44:54.520147 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.520095    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x72qg\" (UniqueName: \"kubernetes.io/projected/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-kube-api-access-x72qg\") pod \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") "
Jul 28 01:44:54.520383 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.520156    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-kserve-provision-location\") pod \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") "
Jul 28 01:44:54.520383 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.520197    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-tls-certs\") pod \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\" (UID: \"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a\") "
Jul 28 01:44:54.520383 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.520310    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" (UID: "ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:44:54.520545 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.520489    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:44:54.520729 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.520697    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-home" (OuterVolumeSpecName: "home") pod "ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" (UID: "ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:44:54.522463 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.522420    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-kube-api-access-x72qg" (OuterVolumeSpecName: "kube-api-access-x72qg") pod "ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" (UID: "ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a"). InnerVolumeSpecName "kube-api-access-x72qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:44:54.522899 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.522880    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-dshm" (OuterVolumeSpecName: "dshm") pod "ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" (UID: "ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:44:54.522970 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.522907    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" (UID: "ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:44:54.538486 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.538458    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" (UID: "ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:44:54.576046 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.576012    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" (UID: "ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:44:54.596182 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.596151    2578 generic.go:356] "Generic (PLEG): container finished" podID="ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" containerID="daea4507c419d5084592fd75908e19da62a9a40ca9c466dbd68ff1b53fa41701" exitCode=0
Jul 28 01:44:54.596297 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.596198    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 01:44:54.596297 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.596209    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a","Type":"ContainerDied","Data":"daea4507c419d5084592fd75908e19da62a9a40ca9c466dbd68ff1b53fa41701"}
Jul 28 01:44:54.596297 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.596251    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a","Type":"ContainerDied","Data":"5c54882c4572783990b9c33f1338fc0d3aace01b5d2753be8c8b4f9f6ee5b55f"}
Jul 28 01:44:54.596297 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.596273    2578 scope.go:117] "RemoveContainer" containerID="daea4507c419d5084592fd75908e19da62a9a40ca9c466dbd68ff1b53fa41701"
Jul 28 01:44:54.605397 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.605383    2578 scope.go:117] "RemoveContainer" containerID="15fe15daa2aaed556038a58a9aea191b3babe19c35d250bd38fe464d64df2133"
Jul 28 01:44:54.612554 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.612539    2578 scope.go:117] "RemoveContainer" containerID="daea4507c419d5084592fd75908e19da62a9a40ca9c466dbd68ff1b53fa41701"
Jul 28 01:44:54.612778 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:44:54.612761    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daea4507c419d5084592fd75908e19da62a9a40ca9c466dbd68ff1b53fa41701\": container with ID starting with daea4507c419d5084592fd75908e19da62a9a40ca9c466dbd68ff1b53fa41701 not found: ID does not exist" containerID="daea4507c419d5084592fd75908e19da62a9a40ca9c466dbd68ff1b53fa41701"
Jul 28 01:44:54.612830 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.612785    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daea4507c419d5084592fd75908e19da62a9a40ca9c466dbd68ff1b53fa41701"} err="failed to get container status \"daea4507c419d5084592fd75908e19da62a9a40ca9c466dbd68ff1b53fa41701\": rpc error: code = NotFound desc = could not find container \"daea4507c419d5084592fd75908e19da62a9a40ca9c466dbd68ff1b53fa41701\": container with ID starting with daea4507c419d5084592fd75908e19da62a9a40ca9c466dbd68ff1b53fa41701 not found: ID does not exist"
Jul 28 01:44:54.612830 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.612802    2578 scope.go:117] "RemoveContainer" containerID="15fe15daa2aaed556038a58a9aea191b3babe19c35d250bd38fe464d64df2133"
Jul 28 01:44:54.613007 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:44:54.612991    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15fe15daa2aaed556038a58a9aea191b3babe19c35d250bd38fe464d64df2133\": container with ID starting with 15fe15daa2aaed556038a58a9aea191b3babe19c35d250bd38fe464d64df2133 not found: ID does not exist" containerID="15fe15daa2aaed556038a58a9aea191b3babe19c35d250bd38fe464d64df2133"
Jul 28 01:44:54.613048 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.613010    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15fe15daa2aaed556038a58a9aea191b3babe19c35d250bd38fe464d64df2133"} err="failed to get container status \"15fe15daa2aaed556038a58a9aea191b3babe19c35d250bd38fe464d64df2133\": rpc error: code = NotFound desc = could not find container \"15fe15daa2aaed556038a58a9aea191b3babe19c35d250bd38fe464d64df2133\": container with ID starting with 15fe15daa2aaed556038a58a9aea191b3babe19c35d250bd38fe464d64df2133 not found: ID does not exist"
Jul 28 01:44:54.618453 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.618433    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 01:44:54.621158 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.621136    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:44:54.621158 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.621155    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:44:54.621279 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.621163    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:44:54.621279 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.621171    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x72qg\" (UniqueName: \"kubernetes.io/projected/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-kube-api-access-x72qg\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:44:54.621279 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.621180    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-kserve-provision-location\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:44:54.621279 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.621189    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:44:54.624769 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.624749    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 01:44:54.985449 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:44:54.985375    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" path="/var/lib/kubelet/pods/ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a/volumes"
Jul 28 01:45:06.651213 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.651184    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"]
Jul 28 01:45:06.651619 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.651488    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" containerName="storage-initializer"
Jul 28 01:45:06.651619 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.651499    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" containerName="storage-initializer"
Jul 28 01:45:06.651619 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.651512    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" containerName="main"
Jul 28 01:45:06.651619 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.651519    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" containerName="main"
Jul 28 01:45:06.651619 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.651573    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab8e1ce4-ec42-4b66-bdcd-2e2976fb066a" containerName="main"
Jul 28 01:45:06.655245 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.655227    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.657950 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.657931    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-48kwd\""
Jul 28 01:45:06.658095 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.658078    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 28 01:45:06.658314 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.658294    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Jul 28 01:45:06.658314 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.658308    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Jul 28 01:45:06.659066 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.659049    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Jul 28 01:45:06.670068 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.670043    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"]
Jul 28 01:45:06.816367 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.816318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-model-cache\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.816504 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.816375    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-dshm\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.816504 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.816436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-home\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.816504 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.816464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-tmp-dir\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.816504 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.816497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dskz5\" (UniqueName: \"kubernetes.io/projected/82da551d-6692-4990-a6e0-a2306a9e932b-kube-api-access-dskz5\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.816647 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.816520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.816647 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.816578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82da551d-6692-4990-a6e0-a2306a9e932b-tls-certs\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.917941 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.917853    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.917941 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.917897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82da551d-6692-4990-a6e0-a2306a9e932b-tls-certs\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.917941 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.917916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-model-cache\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.917941 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.917936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-dshm\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.918233 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.917971    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-home\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.918233 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.917995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-tmp-dir\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.918233 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.918023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dskz5\" (UniqueName: \"kubernetes.io/projected/82da551d-6692-4990-a6e0-a2306a9e932b-kube-api-access-dskz5\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.918403 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.918283    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.918403 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.918325    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-model-cache\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.918403 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.918395    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-home\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.918558 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.918525    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-tmp-dir\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.920232 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.920210    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-dshm\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.920587 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.920569    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82da551d-6692-4990-a6e0-a2306a9e932b-tls-certs\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.926867 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.926847    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dskz5\" (UniqueName: \"kubernetes.io/projected/82da551d-6692-4990-a6e0-a2306a9e932b-kube-api-access-dskz5\") pod \"auth-invalid-token-test-kserve-5ff8967c79-bp7fv\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:06.965320 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:06.965294    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:07.096340 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:07.096265    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"]
Jul 28 01:45:07.098141 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:45:07.098112    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82da551d_6692_4990_a6e0_a2306a9e932b.slice/crio-d60a9206bbf2a7817d4df3aa89f2e1b6de032d34071ff365c14ecddb9b3ac07f WatchSource:0}: Error finding container d60a9206bbf2a7817d4df3aa89f2e1b6de032d34071ff365c14ecddb9b3ac07f: Status 404 returned error can't find the container with id d60a9206bbf2a7817d4df3aa89f2e1b6de032d34071ff365c14ecddb9b3ac07f
Jul 28 01:45:07.654261 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:07.654221    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" event={"ID":"82da551d-6692-4990-a6e0-a2306a9e932b","Type":"ContainerStarted","Data":"422d86e51eb4d23799b7523be971866217ff7911fc864eb44f0cdbc9a0edd79e"}
Jul 28 01:45:07.654675 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:07.654270    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" event={"ID":"82da551d-6692-4990-a6e0-a2306a9e932b","Type":"ContainerStarted","Data":"d60a9206bbf2a7817d4df3aa89f2e1b6de032d34071ff365c14ecddb9b3ac07f"}
Jul 28 01:45:13.681235 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:13.681200    2578 generic.go:356] "Generic (PLEG): container finished" podID="82da551d-6692-4990-a6e0-a2306a9e932b" containerID="422d86e51eb4d23799b7523be971866217ff7911fc864eb44f0cdbc9a0edd79e" exitCode=0
Jul 28 01:45:13.681663 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:13.681286    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" event={"ID":"82da551d-6692-4990-a6e0-a2306a9e932b","Type":"ContainerDied","Data":"422d86e51eb4d23799b7523be971866217ff7911fc864eb44f0cdbc9a0edd79e"}
Jul 28 01:45:14.689116 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:14.689079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" event={"ID":"82da551d-6692-4990-a6e0-a2306a9e932b","Type":"ContainerStarted","Data":"66666c517e152492b659b00e239b698d77381bc946bdc01cfd9326623b024091"}
Jul 28 01:45:14.714669 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:14.714607    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" podStartSLOduration=2.297453039 podStartE2EDuration="8.714590603s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.417137564s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:45:14.714019974 +0000 UTC m=+1158.291477591" watchObservedRunningTime="2026-07-28 01:45:14.714590603 +0000 UTC m=+1158.292048220"
Jul 28 01:45:14.966233 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:14.966131    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:16.966240 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:16.966211    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:45:16.967931 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:16.967903    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" podUID="82da551d-6692-4990-a6e0-a2306a9e932b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 28 01:45:26.965961 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:26.965846    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" podUID="82da551d-6692-4990-a6e0-a2306a9e932b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 28 01:45:36.966246 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:36.966202    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" podUID="82da551d-6692-4990-a6e0-a2306a9e932b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 28 01:45:46.966036 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:46.965990    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" podUID="82da551d-6692-4990-a6e0-a2306a9e932b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 28 01:45:56.966724 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:45:56.966687    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" podUID="82da551d-6692-4990-a6e0-a2306a9e932b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 28 01:46:06.966046 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:46:06.965998    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" podUID="82da551d-6692-4990-a6e0-a2306a9e932b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 28 01:46:16.966858 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:46:16.966804    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" podUID="82da551d-6692-4990-a6e0-a2306a9e932b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 28 01:46:26.966351 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:46:26.966295    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" podUID="82da551d-6692-4990-a6e0-a2306a9e932b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 28 01:46:36.966358 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:46:36.966301    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" podUID="82da551d-6692-4990-a6e0-a2306a9e932b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 28 01:46:46.966771 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:46:46.966718    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" podUID="82da551d-6692-4990-a6e0-a2306a9e932b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 28 01:46:56.966735 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:46:56.966651    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" podUID="82da551d-6692-4990-a6e0-a2306a9e932b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 28 01:47:06.976072 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:06.976034    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:47:06.990990 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:06.990964    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:47:08.759966 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:08.759935    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"]
Jul 28 01:47:23.781743 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:23.781699    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" podUID="82da551d-6692-4990-a6e0-a2306a9e932b" containerName="main" containerID="cri-o://66666c517e152492b659b00e239b698d77381bc946bdc01cfd9326623b024091" gracePeriod=45
Jul 28 01:47:23.977535 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:23.977488    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" podUID="82da551d-6692-4990-a6e0-a2306a9e932b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": dial tcp 10.134.0.29:8000: connect: connection refused"
Jul 28 01:47:24.521086 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.521063    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:47:24.557030 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.556946    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-tmp-dir\") pod \"82da551d-6692-4990-a6e0-a2306a9e932b\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") "
Jul 28 01:47:24.557030 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.556981    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-dshm\") pod \"82da551d-6692-4990-a6e0-a2306a9e932b\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") "
Jul 28 01:47:24.557030 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.557003    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-kserve-provision-location\") pod \"82da551d-6692-4990-a6e0-a2306a9e932b\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") "
Jul 28 01:47:24.557030 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.557020    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-model-cache\") pod \"82da551d-6692-4990-a6e0-a2306a9e932b\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") "
Jul 28 01:47:24.557372 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.557041    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-home\") pod \"82da551d-6692-4990-a6e0-a2306a9e932b\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") "
Jul 28 01:47:24.557372 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.557077    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dskz5\" (UniqueName: \"kubernetes.io/projected/82da551d-6692-4990-a6e0-a2306a9e932b-kube-api-access-dskz5\") pod \"82da551d-6692-4990-a6e0-a2306a9e932b\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") "
Jul 28 01:47:24.557372 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.557125    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82da551d-6692-4990-a6e0-a2306a9e932b-tls-certs\") pod \"82da551d-6692-4990-a6e0-a2306a9e932b\" (UID: \"82da551d-6692-4990-a6e0-a2306a9e932b\") "
Jul 28 01:47:24.557537 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.557464    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-model-cache" (OuterVolumeSpecName: "model-cache") pod "82da551d-6692-4990-a6e0-a2306a9e932b" (UID: "82da551d-6692-4990-a6e0-a2306a9e932b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:47:24.557988 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.557944    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-home" (OuterVolumeSpecName: "home") pod "82da551d-6692-4990-a6e0-a2306a9e932b" (UID: "82da551d-6692-4990-a6e0-a2306a9e932b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:47:24.559722 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.559691    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-dshm" (OuterVolumeSpecName: "dshm") pod "82da551d-6692-4990-a6e0-a2306a9e932b" (UID: "82da551d-6692-4990-a6e0-a2306a9e932b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:47:24.559958 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.559928    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82da551d-6692-4990-a6e0-a2306a9e932b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82da551d-6692-4990-a6e0-a2306a9e932b" (UID: "82da551d-6692-4990-a6e0-a2306a9e932b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:47:24.559958 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.559937    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82da551d-6692-4990-a6e0-a2306a9e932b-kube-api-access-dskz5" (OuterVolumeSpecName: "kube-api-access-dskz5") pod "82da551d-6692-4990-a6e0-a2306a9e932b" (UID: "82da551d-6692-4990-a6e0-a2306a9e932b"). InnerVolumeSpecName "kube-api-access-dskz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:47:24.576379 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.576323    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "82da551d-6692-4990-a6e0-a2306a9e932b" (UID: "82da551d-6692-4990-a6e0-a2306a9e932b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:47:24.621777 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.621735    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "82da551d-6692-4990-a6e0-a2306a9e932b" (UID: "82da551d-6692-4990-a6e0-a2306a9e932b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:47:24.658673 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.658640    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:47:24.658673 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.658671    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:47:24.658860 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.658681    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-kserve-provision-location\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:47:24.658860 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.658693    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:47:24.658860 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.658702    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82da551d-6692-4990-a6e0-a2306a9e932b-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:47:24.658860 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.658709    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dskz5\" (UniqueName: \"kubernetes.io/projected/82da551d-6692-4990-a6e0-a2306a9e932b-kube-api-access-dskz5\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:47:24.658860 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:24.658717    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82da551d-6692-4990-a6e0-a2306a9e932b-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:47:25.271022 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:25.270995    2578 generic.go:356] "Generic (PLEG): container finished" podID="82da551d-6692-4990-a6e0-a2306a9e932b" containerID="66666c517e152492b659b00e239b698d77381bc946bdc01cfd9326623b024091" exitCode=0
Jul 28 01:47:25.271466 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:25.271051    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"
Jul 28 01:47:25.271466 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:25.271077    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" event={"ID":"82da551d-6692-4990-a6e0-a2306a9e932b","Type":"ContainerDied","Data":"66666c517e152492b659b00e239b698d77381bc946bdc01cfd9326623b024091"}
Jul 28 01:47:25.271466 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:25.271134    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv" event={"ID":"82da551d-6692-4990-a6e0-a2306a9e932b","Type":"ContainerDied","Data":"d60a9206bbf2a7817d4df3aa89f2e1b6de032d34071ff365c14ecddb9b3ac07f"}
Jul 28 01:47:25.271466 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:25.271156    2578 scope.go:117] "RemoveContainer" containerID="66666c517e152492b659b00e239b698d77381bc946bdc01cfd9326623b024091"
Jul 28 01:47:25.279485 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:25.279469    2578 scope.go:117] "RemoveContainer" containerID="422d86e51eb4d23799b7523be971866217ff7911fc864eb44f0cdbc9a0edd79e"
Jul 28 01:47:25.287217 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:25.287193    2578 scope.go:117] "RemoveContainer" containerID="66666c517e152492b659b00e239b698d77381bc946bdc01cfd9326623b024091"
Jul 28 01:47:25.287469 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:47:25.287447    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66666c517e152492b659b00e239b698d77381bc946bdc01cfd9326623b024091\": container with ID starting with 66666c517e152492b659b00e239b698d77381bc946bdc01cfd9326623b024091 not found: ID does not exist" containerID="66666c517e152492b659b00e239b698d77381bc946bdc01cfd9326623b024091"
Jul 28 01:47:25.287523 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:25.287478    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66666c517e152492b659b00e239b698d77381bc946bdc01cfd9326623b024091"} err="failed to get container status \"66666c517e152492b659b00e239b698d77381bc946bdc01cfd9326623b024091\": rpc error: code = NotFound desc = could not find container \"66666c517e152492b659b00e239b698d77381bc946bdc01cfd9326623b024091\": container with ID starting with 66666c517e152492b659b00e239b698d77381bc946bdc01cfd9326623b024091 not found: ID does not exist"
Jul 28 01:47:25.287523 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:25.287494    2578 scope.go:117] "RemoveContainer" containerID="422d86e51eb4d23799b7523be971866217ff7911fc864eb44f0cdbc9a0edd79e"
Jul 28 01:47:25.287709 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:47:25.287689    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"422d86e51eb4d23799b7523be971866217ff7911fc864eb44f0cdbc9a0edd79e\": container with ID starting with 422d86e51eb4d23799b7523be971866217ff7911fc864eb44f0cdbc9a0edd79e not found: ID does not exist" containerID="422d86e51eb4d23799b7523be971866217ff7911fc864eb44f0cdbc9a0edd79e"
Jul 28 01:47:25.287748 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:25.287715    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"422d86e51eb4d23799b7523be971866217ff7911fc864eb44f0cdbc9a0edd79e"} err="failed to get container status \"422d86e51eb4d23799b7523be971866217ff7911fc864eb44f0cdbc9a0edd79e\": rpc error: code = NotFound desc = could not find container \"422d86e51eb4d23799b7523be971866217ff7911fc864eb44f0cdbc9a0edd79e\": container with ID starting with 422d86e51eb4d23799b7523be971866217ff7911fc864eb44f0cdbc9a0edd79e not found: ID does not exist"
Jul 28 01:47:25.292832 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:25.292813    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"]
Jul 28 01:47:25.296742 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:25.296720    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5ff8967c79-bp7fv"]
Jul 28 01:47:26.985963 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:26.985928    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82da551d-6692-4990-a6e0-a2306a9e932b" path="/var/lib/kubelet/pods/82da551d-6692-4990-a6e0-a2306a9e932b/volumes"
Jul 28 01:47:37.946156 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:37.946125    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"]
Jul 28 01:47:37.946562 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:37.946423    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82da551d-6692-4990-a6e0-a2306a9e932b" containerName="main"
Jul 28 01:47:37.946562 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:37.946435    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="82da551d-6692-4990-a6e0-a2306a9e932b" containerName="main"
Jul 28 01:47:37.946562 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:37.946448    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82da551d-6692-4990-a6e0-a2306a9e932b" containerName="storage-initializer"
Jul 28 01:47:37.946562 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:37.946454    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="82da551d-6692-4990-a6e0-a2306a9e932b" containerName="storage-initializer"
Jul 28 01:47:37.946562 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:37.946503    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="82da551d-6692-4990-a6e0-a2306a9e932b" containerName="main"
Jul 28 01:47:37.949481 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:37.949463    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:37.952491 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:37.952473    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-8chgm\""
Jul 28 01:47:37.952676 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:37.952663    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Jul 28 01:47:37.952727 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:37.952711    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Jul 28 01:47:37.953733 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:37.953713    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 28 01:47:37.953733 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:37.953725    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Jul 28 01:47:37.961140 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:37.961119    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"]
Jul 28 01:47:38.053633 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.053603    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-tmp-dir\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.053852 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.053690    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qx8n\" (UniqueName: \"kubernetes.io/projected/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-kube-api-access-6qx8n\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.053852 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.053726    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-kserve-provision-location\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.053852 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.053787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-tls-certs\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.053852 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.053840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-home\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.054010 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.053866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-model-cache\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.054010 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.053891    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-dshm\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.155025 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.154996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qx8n\" (UniqueName: \"kubernetes.io/projected/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-kube-api-access-6qx8n\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.155025 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.155026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-kserve-provision-location\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.155202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.155049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-tls-certs\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.155202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.155103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-home\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.155202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.155129    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-model-cache\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.155202 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.155150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-dshm\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.155414 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.155215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-tmp-dir\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.155525 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.155505    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-home\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.155599 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.155551    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-model-cache\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.155599 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.155589    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-kserve-provision-location\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.155681 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.155657    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-tmp-dir\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.157472 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.157453    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-dshm\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.157774 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.157757    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-tls-certs\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.166418 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.166398    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qx8n\" (UniqueName: \"kubernetes.io/projected/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-kube-api-access-6qx8n\") pod \"auth-disabled-test-kserve-6f468f6548-2cdcp\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.259042 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.258980    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:38.392948 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:38.392858    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"]
Jul 28 01:47:38.395582 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:47:38.395551    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedaae7a6_cb1f_4cfd_92df_2c43a447e52f.slice/crio-aee70ac05a7b45f956ca836753dc9dda3e1a544a856c8c311d9417ad6c810dfb WatchSource:0}: Error finding container aee70ac05a7b45f956ca836753dc9dda3e1a544a856c8c311d9417ad6c810dfb: Status 404 returned error can't find the container with id aee70ac05a7b45f956ca836753dc9dda3e1a544a856c8c311d9417ad6c810dfb
Jul 28 01:47:39.337792 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:39.337751    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" event={"ID":"edaae7a6-cb1f-4cfd-92df-2c43a447e52f","Type":"ContainerStarted","Data":"45ceb10e2e7e04114fca5317cd676bf18087ac9000878496c0a2f2266334def8"}
Jul 28 01:47:39.338397 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:39.337810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" event={"ID":"edaae7a6-cb1f-4cfd-92df-2c43a447e52f","Type":"ContainerStarted","Data":"aee70ac05a7b45f956ca836753dc9dda3e1a544a856c8c311d9417ad6c810dfb"}
Jul 28 01:47:45.365535 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:45.365501    2578 generic.go:356] "Generic (PLEG): container finished" podID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerID="45ceb10e2e7e04114fca5317cd676bf18087ac9000878496c0a2f2266334def8" exitCode=0
Jul 28 01:47:45.365931 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:45.365575    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" event={"ID":"edaae7a6-cb1f-4cfd-92df-2c43a447e52f","Type":"ContainerDied","Data":"45ceb10e2e7e04114fca5317cd676bf18087ac9000878496c0a2f2266334def8"}
Jul 28 01:47:46.371946 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:46.371907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" event={"ID":"edaae7a6-cb1f-4cfd-92df-2c43a447e52f","Type":"ContainerStarted","Data":"2cf0495d60582589e98418182981d0c207293b8592e47e97ae430fbb07c35e9d"}
Jul 28 01:47:46.395577 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:46.395511    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" podStartSLOduration=3.179662189 podStartE2EDuration="9.395486984s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.215824795s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:47:46.392126398 +0000 UTC m=+1309.969584017" watchObservedRunningTime="2026-07-28 01:47:46.395486984 +0000 UTC m=+1309.972944602"
Jul 28 01:47:47.259241 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:47.259205    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:48.259740 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:48.259706    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:47:48.261323 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:48.261288    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" podUID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 28 01:47:58.260268 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:47:58.260216    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" podUID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 28 01:48:08.260296 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:08.260250    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" podUID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 28 01:48:18.259816 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:18.259722    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" podUID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 28 01:48:28.260353 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:28.260289    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" podUID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 28 01:48:38.260373 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:38.260306    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" podUID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 28 01:48:48.259985 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:48.259936    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" podUID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 28 01:48:54.226564 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:54.226531    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s"]
Jul 28 01:48:54.229250 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:54.229232    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s"
Jul 28 01:48:54.234270 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:54.234245    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 28 01:48:54.234456 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:54.234245    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-tk5n9\""
Jul 28 01:48:54.234456 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:54.234250    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 28 01:48:54.234633 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:54.234614    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 28 01:48:54.234845 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:54.234830    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-vvkxm\""
Jul 28 01:48:54.239843 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:54.239819    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s"]
Jul 28 01:48:54.282899 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:54.282857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b25f69-c3eb-4a06-a100-2a7e05e0a700-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s\" (UID: \"95b25f69-c3eb-4a06-a100-2a7e05e0a700\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s"
Jul 28 01:48:54.283073 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:54.282920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msf86\" (UniqueName: \"kubernetes.io/projected/95b25f69-c3eb-4a06-a100-2a7e05e0a700-kube-api-access-msf86\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s\" (UID: \"95b25f69-c3eb-4a06-a100-2a7e05e0a700\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s"
Jul 28 01:48:54.383920 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:54.383879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b25f69-c3eb-4a06-a100-2a7e05e0a700-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s\" (UID: \"95b25f69-c3eb-4a06-a100-2a7e05e0a700\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s"
Jul 28 01:48:54.384114 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:54.383952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msf86\" (UniqueName: \"kubernetes.io/projected/95b25f69-c3eb-4a06-a100-2a7e05e0a700-kube-api-access-msf86\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s\" (UID: \"95b25f69-c3eb-4a06-a100-2a7e05e0a700\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s"
Jul 28 01:48:54.386721 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:54.386687    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b25f69-c3eb-4a06-a100-2a7e05e0a700-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s\" (UID: \"95b25f69-c3eb-4a06-a100-2a7e05e0a700\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s"
Jul 28 01:48:54.394603 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:54.394577    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msf86\" (UniqueName: \"kubernetes.io/projected/95b25f69-c3eb-4a06-a100-2a7e05e0a700-kube-api-access-msf86\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s\" (UID: \"95b25f69-c3eb-4a06-a100-2a7e05e0a700\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s"
Jul 28 01:48:54.541287 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:54.541246    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s"
Jul 28 01:48:54.675906 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:54.675883    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s"]
Jul 28 01:48:54.678252 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:48:54.678224    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b25f69_c3eb_4a06_a100_2a7e05e0a700.slice/crio-6928722d9476634129e3d593c9910f340d7d26f2b29653a5cb262a8ae509f120 WatchSource:0}: Error finding container 6928722d9476634129e3d593c9910f340d7d26f2b29653a5cb262a8ae509f120: Status 404 returned error can't find the container with id 6928722d9476634129e3d593c9910f340d7d26f2b29653a5cb262a8ae509f120
Jul 28 01:48:54.680068 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:54.680047    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:48:55.680367 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:55.680307    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s" event={"ID":"95b25f69-c3eb-4a06-a100-2a7e05e0a700","Type":"ContainerStarted","Data":"6928722d9476634129e3d593c9910f340d7d26f2b29653a5cb262a8ae509f120"}
Jul 28 01:48:57.693037 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:57.693005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s" event={"ID":"95b25f69-c3eb-4a06-a100-2a7e05e0a700","Type":"ContainerStarted","Data":"9da13a3b44a3530499eb977d90dde5c986fccff44562645414b8503fcb6c25fa"}
Jul 28 01:48:57.693543 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:57.693079    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s"
Jul 28 01:48:57.713503 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:57.713455    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s" podStartSLOduration=1.385072642 podStartE2EDuration="3.713440527s" totalImagesPullingTime="2.328367885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:48:57.712487549 +0000 UTC m=+1381.289945174" watchObservedRunningTime="2026-07-28 01:48:57.713440527 +0000 UTC m=+1381.290898144"
Jul 28 01:48:58.259520 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:48:58.259477    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" podUID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 28 01:49:08.259929 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:08.259877    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" podUID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 28 01:49:18.259863 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:18.259821    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" podUID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 28 01:49:27.592213 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:27.592172    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s"]
Jul 28 01:49:27.594111 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:27.594073    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s" podUID="95b25f69-c3eb-4a06-a100-2a7e05e0a700" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:49:28.260118 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:28.260074    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" podUID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 28 01:49:37.594150 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:37.594113    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s" podUID="95b25f69-c3eb-4a06-a100-2a7e05e0a700" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 01:49:38.270123 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:38.270097    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:49:38.277980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:38.277957    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:49:42.611992 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:42.611939    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s" podUID="95b25f69-c3eb-4a06-a100-2a7e05e0a700" containerName="main" containerID="cri-o://9da13a3b44a3530499eb977d90dde5c986fccff44562645414b8503fcb6c25fa" gracePeriod=45
Jul 28 01:49:42.877916 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:42.877895    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s"
Jul 28 01:49:42.915549 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:42.915491    2578 generic.go:356] "Generic (PLEG): container finished" podID="95b25f69-c3eb-4a06-a100-2a7e05e0a700" containerID="9da13a3b44a3530499eb977d90dde5c986fccff44562645414b8503fcb6c25fa" exitCode=0
Jul 28 01:49:42.915762 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:42.915650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s" event={"ID":"95b25f69-c3eb-4a06-a100-2a7e05e0a700","Type":"ContainerDied","Data":"9da13a3b44a3530499eb977d90dde5c986fccff44562645414b8503fcb6c25fa"}
Jul 28 01:49:42.915762 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:42.915753    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s" event={"ID":"95b25f69-c3eb-4a06-a100-2a7e05e0a700","Type":"ContainerDied","Data":"6928722d9476634129e3d593c9910f340d7d26f2b29653a5cb262a8ae509f120"}
Jul 28 01:49:42.915914 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:42.915806    2578 scope.go:117] "RemoveContainer" containerID="9da13a3b44a3530499eb977d90dde5c986fccff44562645414b8503fcb6c25fa"
Jul 28 01:49:42.916046 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:42.916026    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s"
Jul 28 01:49:42.932204 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:42.932150    2578 scope.go:117] "RemoveContainer" containerID="9da13a3b44a3530499eb977d90dde5c986fccff44562645414b8503fcb6c25fa"
Jul 28 01:49:42.932760 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:49:42.932727    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9da13a3b44a3530499eb977d90dde5c986fccff44562645414b8503fcb6c25fa\": container with ID starting with 9da13a3b44a3530499eb977d90dde5c986fccff44562645414b8503fcb6c25fa not found: ID does not exist" containerID="9da13a3b44a3530499eb977d90dde5c986fccff44562645414b8503fcb6c25fa"
Jul 28 01:49:42.932965 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:42.932925    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9da13a3b44a3530499eb977d90dde5c986fccff44562645414b8503fcb6c25fa"} err="failed to get container status \"9da13a3b44a3530499eb977d90dde5c986fccff44562645414b8503fcb6c25fa\": rpc error: code = NotFound desc = could not find container \"9da13a3b44a3530499eb977d90dde5c986fccff44562645414b8503fcb6c25fa\": container with ID starting with 9da13a3b44a3530499eb977d90dde5c986fccff44562645414b8503fcb6c25fa not found: ID does not exist"
Jul 28 01:49:42.994276 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:42.994243    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msf86\" (UniqueName: \"kubernetes.io/projected/95b25f69-c3eb-4a06-a100-2a7e05e0a700-kube-api-access-msf86\") pod \"95b25f69-c3eb-4a06-a100-2a7e05e0a700\" (UID: \"95b25f69-c3eb-4a06-a100-2a7e05e0a700\") "
Jul 28 01:49:42.994514 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:42.994433    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b25f69-c3eb-4a06-a100-2a7e05e0a700-tls-certs\") pod \"95b25f69-c3eb-4a06-a100-2a7e05e0a700\" (UID: \"95b25f69-c3eb-4a06-a100-2a7e05e0a700\") "
Jul 28 01:49:42.997940 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:42.997905    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b25f69-c3eb-4a06-a100-2a7e05e0a700-kube-api-access-msf86" (OuterVolumeSpecName: "kube-api-access-msf86") pod "95b25f69-c3eb-4a06-a100-2a7e05e0a700" (UID: "95b25f69-c3eb-4a06-a100-2a7e05e0a700"). InnerVolumeSpecName "kube-api-access-msf86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:49:42.998138 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:42.997909    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95b25f69-c3eb-4a06-a100-2a7e05e0a700-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95b25f69-c3eb-4a06-a100-2a7e05e0a700" (UID: "95b25f69-c3eb-4a06-a100-2a7e05e0a700"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:49:43.095105 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:43.095075    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b25f69-c3eb-4a06-a100-2a7e05e0a700-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:49:43.095105 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:43.095101    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msf86\" (UniqueName: \"kubernetes.io/projected/95b25f69-c3eb-4a06-a100-2a7e05e0a700-kube-api-access-msf86\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:49:43.197902 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:43.197815    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"]
Jul 28 01:49:43.240652 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:43.240624    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s"]
Jul 28 01:49:43.246146 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:43.246122    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfc4b7s"]
Jul 28 01:49:44.985344 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:44.985303    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95b25f69-c3eb-4a06-a100-2a7e05e0a700" path="/var/lib/kubelet/pods/95b25f69-c3eb-4a06-a100-2a7e05e0a700/volumes"
Jul 28 01:49:58.220121 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.220069    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" podUID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerName="main" containerID="cri-o://2cf0495d60582589e98418182981d0c207293b8592e47e97ae430fbb07c35e9d" gracePeriod=45
Jul 28 01:49:58.271301 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.271255    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" podUID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 28 01:49:58.866952 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.866930    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:49:58.928015 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.927981    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qx8n\" (UniqueName: \"kubernetes.io/projected/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-kube-api-access-6qx8n\") pod \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") "
Jul 28 01:49:58.928182 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.928059    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-tls-certs\") pod \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") "
Jul 28 01:49:58.928182 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.928087    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-model-cache\") pod \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") "
Jul 28 01:49:58.928182 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.928124    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-kserve-provision-location\") pod \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") "
Jul 28 01:49:58.928182 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.928145    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-dshm\") pod \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") "
Jul 28 01:49:58.928182 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.928175    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-tmp-dir\") pod \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") "
Jul 28 01:49:58.928467 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.928196    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-home\") pod \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\" (UID: \"edaae7a6-cb1f-4cfd-92df-2c43a447e52f\") "
Jul 28 01:49:58.928467 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.928318    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-model-cache" (OuterVolumeSpecName: "model-cache") pod "edaae7a6-cb1f-4cfd-92df-2c43a447e52f" (UID: "edaae7a6-cb1f-4cfd-92df-2c43a447e52f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:49:58.928578 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.928488    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:49:58.928931 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.928909    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-home" (OuterVolumeSpecName: "home") pod "edaae7a6-cb1f-4cfd-92df-2c43a447e52f" (UID: "edaae7a6-cb1f-4cfd-92df-2c43a447e52f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:49:58.930799 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.930779    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-kube-api-access-6qx8n" (OuterVolumeSpecName: "kube-api-access-6qx8n") pod "edaae7a6-cb1f-4cfd-92df-2c43a447e52f" (UID: "edaae7a6-cb1f-4cfd-92df-2c43a447e52f"). InnerVolumeSpecName "kube-api-access-6qx8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:49:58.930880 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.930853    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "edaae7a6-cb1f-4cfd-92df-2c43a447e52f" (UID: "edaae7a6-cb1f-4cfd-92df-2c43a447e52f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:49:58.930919 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.930880    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-dshm" (OuterVolumeSpecName: "dshm") pod "edaae7a6-cb1f-4cfd-92df-2c43a447e52f" (UID: "edaae7a6-cb1f-4cfd-92df-2c43a447e52f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:49:58.941303 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.941272    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "edaae7a6-cb1f-4cfd-92df-2c43a447e52f" (UID: "edaae7a6-cb1f-4cfd-92df-2c43a447e52f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:49:58.985522 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.985452    2578 generic.go:356] "Generic (PLEG): container finished" podID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerID="2cf0495d60582589e98418182981d0c207293b8592e47e97ae430fbb07c35e9d" exitCode=0
Jul 28 01:49:58.985639 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.985575    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"
Jul 28 01:49:58.986868 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.986841    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" event={"ID":"edaae7a6-cb1f-4cfd-92df-2c43a447e52f","Type":"ContainerDied","Data":"2cf0495d60582589e98418182981d0c207293b8592e47e97ae430fbb07c35e9d"}
Jul 28 01:49:58.986959 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.986889    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp" event={"ID":"edaae7a6-cb1f-4cfd-92df-2c43a447e52f","Type":"ContainerDied","Data":"aee70ac05a7b45f956ca836753dc9dda3e1a544a856c8c311d9417ad6c810dfb"}
Jul 28 01:49:58.986959 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.986911    2578 scope.go:117] "RemoveContainer" containerID="2cf0495d60582589e98418182981d0c207293b8592e47e97ae430fbb07c35e9d"
Jul 28 01:49:58.996275 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:58.996255    2578 scope.go:117] "RemoveContainer" containerID="45ceb10e2e7e04114fca5317cd676bf18087ac9000878496c0a2f2266334def8"
Jul 28 01:49:59.002000 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:59.001975    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "edaae7a6-cb1f-4cfd-92df-2c43a447e52f" (UID: "edaae7a6-cb1f-4cfd-92df-2c43a447e52f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:49:59.003736 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:59.003721    2578 scope.go:117] "RemoveContainer" containerID="2cf0495d60582589e98418182981d0c207293b8592e47e97ae430fbb07c35e9d"
Jul 28 01:49:59.003994 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:49:59.003978    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf0495d60582589e98418182981d0c207293b8592e47e97ae430fbb07c35e9d\": container with ID starting with 2cf0495d60582589e98418182981d0c207293b8592e47e97ae430fbb07c35e9d not found: ID does not exist" containerID="2cf0495d60582589e98418182981d0c207293b8592e47e97ae430fbb07c35e9d"
Jul 28 01:49:59.004042 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:59.004003    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf0495d60582589e98418182981d0c207293b8592e47e97ae430fbb07c35e9d"} err="failed to get container status \"2cf0495d60582589e98418182981d0c207293b8592e47e97ae430fbb07c35e9d\": rpc error: code = NotFound desc = could not find container \"2cf0495d60582589e98418182981d0c207293b8592e47e97ae430fbb07c35e9d\": container with ID starting with 2cf0495d60582589e98418182981d0c207293b8592e47e97ae430fbb07c35e9d not found: ID does not exist"
Jul 28 01:49:59.004042 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:59.004021    2578 scope.go:117] "RemoveContainer" containerID="45ceb10e2e7e04114fca5317cd676bf18087ac9000878496c0a2f2266334def8"
Jul 28 01:49:59.004257 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:49:59.004236    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ceb10e2e7e04114fca5317cd676bf18087ac9000878496c0a2f2266334def8\": container with ID starting with 45ceb10e2e7e04114fca5317cd676bf18087ac9000878496c0a2f2266334def8 not found: ID does not exist" containerID="45ceb10e2e7e04114fca5317cd676bf18087ac9000878496c0a2f2266334def8"
Jul 28 01:49:59.004310 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:59.004268    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ceb10e2e7e04114fca5317cd676bf18087ac9000878496c0a2f2266334def8"} err="failed to get container status \"45ceb10e2e7e04114fca5317cd676bf18087ac9000878496c0a2f2266334def8\": rpc error: code = NotFound desc = could not find container \"45ceb10e2e7e04114fca5317cd676bf18087ac9000878496c0a2f2266334def8\": container with ID starting with 45ceb10e2e7e04114fca5317cd676bf18087ac9000878496c0a2f2266334def8 not found: ID does not exist"
Jul 28 01:49:59.029321 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:59.029300    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-kserve-provision-location\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:49:59.029321 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:59.029320    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:49:59.029436 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:59.029346    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:49:59.029436 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:59.029354    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:49:59.029436 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:59.029362    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qx8n\" (UniqueName: \"kubernetes.io/projected/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-kube-api-access-6qx8n\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:49:59.029436 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:59.029371    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edaae7a6-cb1f-4cfd-92df-2c43a447e52f-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:49:59.307944 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:59.307914    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"]
Jul 28 01:49:59.312072 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:49:59.312048    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-6f468f6548-2cdcp"]
Jul 28 01:50:00.985288 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:00.985257    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" path="/var/lib/kubelet/pods/edaae7a6-cb1f-4cfd-92df-2c43a447e52f/volumes"
Jul 28 01:50:01.846154 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.846122    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"]
Jul 28 01:50:01.846439 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.846425    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerName="main"
Jul 28 01:50:01.846495 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.846440    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerName="main"
Jul 28 01:50:01.846495 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.846451    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerName="storage-initializer"
Jul 28 01:50:01.846495 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.846457    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerName="storage-initializer"
Jul 28 01:50:01.846495 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.846469    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95b25f69-c3eb-4a06-a100-2a7e05e0a700" containerName="main"
Jul 28 01:50:01.846495 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.846475    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="95b25f69-c3eb-4a06-a100-2a7e05e0a700" containerName="main"
Jul 28 01:50:01.846673 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.846540    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="edaae7a6-cb1f-4cfd-92df-2c43a447e52f" containerName="main"
Jul 28 01:50:01.846673 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.846546    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="95b25f69-c3eb-4a06-a100-2a7e05e0a700" containerName="main"
Jul 28 01:50:01.850917 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.850898    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:01.854065 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.854045    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Jul 28 01:50:01.855689 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.855669    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-rksqq\""
Jul 28 01:50:01.855785 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.855712    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Jul 28 01:50:01.855785 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.855715    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 28 01:50:01.861818 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.861797    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"]
Jul 28 01:50:01.952878 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.952846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6bp7s\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:01.952878 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.952882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6bp7s\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:01.953150 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.952906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6bp7s\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:01.953150 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.952922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwdh8\" (UniqueName: \"kubernetes.io/projected/b22f18cb-b039-4321-ac58-fdcd3fe292a7-kube-api-access-bwdh8\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6bp7s\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:01.953150 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.953004    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6bp7s\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:01.953150 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:01.953063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b22f18cb-b039-4321-ac58-fdcd3fe292a7-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6bp7s\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:02.053822 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:02.053787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6bp7s\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:02.054347 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:02.053858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b22f18cb-b039-4321-ac58-fdcd3fe292a7-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6bp7s\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:02.054347 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:02.053938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6bp7s\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:02.054347 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:02.053968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6bp7s\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:02.054347 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:02.054007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6bp7s\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:02.054347 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:02.054031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwdh8\" (UniqueName: \"kubernetes.io/projected/b22f18cb-b039-4321-ac58-fdcd3fe292a7-kube-api-access-bwdh8\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6bp7s\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:02.054347 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:02.054271    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6bp7s\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:02.054590 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:02.054429    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6bp7s\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:02.054590 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:02.054512    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6bp7s\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:02.056535 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:02.056497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6bp7s\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:02.056535 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:02.056530    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b22f18cb-b039-4321-ac58-fdcd3fe292a7-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6bp7s\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:02.067547 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:02.067517    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwdh8\" (UniqueName: \"kubernetes.io/projected/b22f18cb-b039-4321-ac58-fdcd3fe292a7-kube-api-access-bwdh8\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-6bp7s\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:02.161139 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:02.161028    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:02.293613 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:02.293587    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"]
Jul 28 01:50:02.296287 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:50:02.296263    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22f18cb_b039_4321_ac58_fdcd3fe292a7.slice/crio-bd4847c4c7708e29e718ca703e0eefc2c2b210e5c6e9994a163073dadf699cc1 WatchSource:0}: Error finding container bd4847c4c7708e29e718ca703e0eefc2c2b210e5c6e9994a163073dadf699cc1: Status 404 returned error can't find the container with id bd4847c4c7708e29e718ca703e0eefc2c2b210e5c6e9994a163073dadf699cc1
Jul 28 01:50:03.005210 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:03.005180    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s" event={"ID":"b22f18cb-b039-4321-ac58-fdcd3fe292a7","Type":"ContainerStarted","Data":"e61ff11ac17888750bf589cb29d049c3f1635e65dd2ba7188dadbc03897a3595"}
Jul 28 01:50:03.005499 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:03.005229    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s" event={"ID":"b22f18cb-b039-4321-ac58-fdcd3fe292a7","Type":"ContainerStarted","Data":"bd4847c4c7708e29e718ca703e0eefc2c2b210e5c6e9994a163073dadf699cc1"}
Jul 28 01:50:03.027663 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:03.027615    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s" podStartSLOduration=2.027602083 podStartE2EDuration="2.027602083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:50:03.026561025 +0000 UTC m=+1446.604018642" watchObservedRunningTime="2026-07-28 01:50:03.027602083 +0000 UTC m=+1446.605059710"
Jul 28 01:50:03.161431 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:03.161397    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:12.162028 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:12.161992    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:12.174575 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:12.174551    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:13.055919 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:13.055891    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:28.531443 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:28.531410    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc"]
Jul 28 01:50:28.534782 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:28.534761    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc"
Jul 28 01:50:28.538084 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:28.538060    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-kserve-self-signed-certs\""
Jul 28 01:50:28.539414 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:28.539395    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"default-dockercfg-b8br4\""
Jul 28 01:50:28.539521 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:28.539421    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"kube-root-ca.crt\""
Jul 28 01:50:28.539521 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:28.539489    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"openshift-service-ca.crt\""
Jul 28 01:50:28.539521 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:28.539517    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-epp-sa-dockercfg-9fghw\""
Jul 28 01:50:28.544160 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:28.544135    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc"]
Jul 28 01:50:28.567483 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:28.567458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4724367b-3420-49c1-8388-f26595737bd5-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc\" (UID: \"4724367b-3420-49c1-8388-f26595737bd5\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc"
Jul 28 01:50:28.567630 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:28.567490    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgk4v\" (UniqueName: \"kubernetes.io/projected/4724367b-3420-49c1-8388-f26595737bd5-kube-api-access-qgk4v\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc\" (UID: \"4724367b-3420-49c1-8388-f26595737bd5\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc"
Jul 28 01:50:28.667905 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:28.667871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4724367b-3420-49c1-8388-f26595737bd5-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc\" (UID: \"4724367b-3420-49c1-8388-f26595737bd5\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc"
Jul 28 01:50:28.667905 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:28.667907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgk4v\" (UniqueName: \"kubernetes.io/projected/4724367b-3420-49c1-8388-f26595737bd5-kube-api-access-qgk4v\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc\" (UID: \"4724367b-3420-49c1-8388-f26595737bd5\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc"
Jul 28 01:50:28.670322 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:28.670294    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4724367b-3420-49c1-8388-f26595737bd5-tls-certs\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc\" (UID: \"4724367b-3420-49c1-8388-f26595737bd5\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc"
Jul 28 01:50:28.676482 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:28.676459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgk4v\" (UniqueName: \"kubernetes.io/projected/4724367b-3420-49c1-8388-f26595737bd5-kube-api-access-qgk4v\") pod \"autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc\" (UID: \"4724367b-3420-49c1-8388-f26595737bd5\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc"
Jul 28 01:50:28.846491 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:28.846408    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc"
Jul 28 01:50:28.975235 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:28.975208    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc"]
Jul 28 01:50:28.977597 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:50:28.977571    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4724367b_3420_49c1_8388_f26595737bd5.slice/crio-aa75fb276ac0f08e09d337c71a9dfc52cd2c7d6b93b5004dde3d6d8a3c16fd24 WatchSource:0}: Error finding container aa75fb276ac0f08e09d337c71a9dfc52cd2c7d6b93b5004dde3d6d8a3c16fd24: Status 404 returned error can't find the container with id aa75fb276ac0f08e09d337c71a9dfc52cd2c7d6b93b5004dde3d6d8a3c16fd24
Jul 28 01:50:29.116748 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:29.116654    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc" event={"ID":"4724367b-3420-49c1-8388-f26595737bd5","Type":"ContainerStarted","Data":"cfb2c7ba6462e17d7b133de5ebad6eb43d7743769edde2241e523cc5284cdf31"}
Jul 28 01:50:29.116748 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:29.116718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc" event={"ID":"4724367b-3420-49c1-8388-f26595737bd5","Type":"ContainerStarted","Data":"aa75fb276ac0f08e09d337c71a9dfc52cd2c7d6b93b5004dde3d6d8a3c16fd24"}
Jul 28 01:50:29.116982 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:29.116752    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc"
Jul 28 01:50:29.133748 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:29.133693    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc" podStartSLOduration=1.133670864 podStartE2EDuration="1.133670864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:50:29.133045654 +0000 UTC m=+1472.710503271" watchObservedRunningTime="2026-07-28 01:50:29.133670864 +0000 UTC m=+1472.711128481"
Jul 28 01:50:37.886194 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:37.886161    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"]
Jul 28 01:50:37.895451 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:50:37.895416    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:50:37.895451 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:50:37.895451 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:50:37.895451 ip-10-0-139-81 kubenswrapper[2578]:  > containerID="e61ff11ac17888750bf589cb29d049c3f1635e65dd2ba7188dadbc03897a3595" cmd=["/bin/sleep","15"]
Jul 28 01:50:37.895724 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:37.895469    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s" message="" err=<
Jul 28 01:50:37.895724 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:50:37.895724 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:50:37.895724 ip-10-0-139-81 kubenswrapper[2578]:  >
Jul 28 01:50:37.895724 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:50:37.895481    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:50:37.895724 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:50:37.895724 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:50:37.895724 ip-10-0-139-81 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s" podUID="b22f18cb-b039-4321-ac58-fdcd3fe292a7" containerName="main" containerID="cri-o://e61ff11ac17888750bf589cb29d049c3f1635e65dd2ba7188dadbc03897a3595"
Jul 28 01:50:37.895724 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:37.895525    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s" podUID="b22f18cb-b039-4321-ac58-fdcd3fe292a7" containerName="main" containerID="cri-o://e61ff11ac17888750bf589cb29d049c3f1635e65dd2ba7188dadbc03897a3595" gracePeriod=60
Jul 28 01:50:38.045718 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.045676    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s" podUID="b22f18cb-b039-4321-ac58-fdcd3fe292a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 28 01:50:38.159150 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.159048    2578 generic.go:356] "Generic (PLEG): container finished" podID="b22f18cb-b039-4321-ac58-fdcd3fe292a7" containerID="e61ff11ac17888750bf589cb29d049c3f1635e65dd2ba7188dadbc03897a3595" exitCode=0
Jul 28 01:50:38.159150 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.159112    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s" event={"ID":"b22f18cb-b039-4321-ac58-fdcd3fe292a7","Type":"ContainerDied","Data":"e61ff11ac17888750bf589cb29d049c3f1635e65dd2ba7188dadbc03897a3595"}
Jul 28 01:50:38.255318 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.255287    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:38.348125 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.348088    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwdh8\" (UniqueName: \"kubernetes.io/projected/b22f18cb-b039-4321-ac58-fdcd3fe292a7-kube-api-access-bwdh8\") pod \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") "
Jul 28 01:50:38.348353 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.348145    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-model-cache\") pod \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") "
Jul 28 01:50:38.348353 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.348177    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-home\") pod \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") "
Jul 28 01:50:38.348353 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.348227    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-tmp-dir\") pod \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") "
Jul 28 01:50:38.348353 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.348253    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-dshm\") pod \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") "
Jul 28 01:50:38.348353 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.348267    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b22f18cb-b039-4321-ac58-fdcd3fe292a7-tls-certs\") pod \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\" (UID: \"b22f18cb-b039-4321-ac58-fdcd3fe292a7\") "
Jul 28 01:50:38.348619 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.348471    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "b22f18cb-b039-4321-ac58-fdcd3fe292a7" (UID: "b22f18cb-b039-4321-ac58-fdcd3fe292a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:50:38.348619 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.348489    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b22f18cb-b039-4321-ac58-fdcd3fe292a7" (UID: "b22f18cb-b039-4321-ac58-fdcd3fe292a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:50:38.348619 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.348514    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-home" (OuterVolumeSpecName: "home") pod "b22f18cb-b039-4321-ac58-fdcd3fe292a7" (UID: "b22f18cb-b039-4321-ac58-fdcd3fe292a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:50:38.348619 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.348573    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:50:38.348619 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.348588    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:50:38.348619 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.348603    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:50:38.350554 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.350533    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-dshm" (OuterVolumeSpecName: "dshm") pod "b22f18cb-b039-4321-ac58-fdcd3fe292a7" (UID: "b22f18cb-b039-4321-ac58-fdcd3fe292a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:50:38.350655 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.350583    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22f18cb-b039-4321-ac58-fdcd3fe292a7-kube-api-access-bwdh8" (OuterVolumeSpecName: "kube-api-access-bwdh8") pod "b22f18cb-b039-4321-ac58-fdcd3fe292a7" (UID: "b22f18cb-b039-4321-ac58-fdcd3fe292a7"). InnerVolumeSpecName "kube-api-access-bwdh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:50:38.350655 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.350630    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b22f18cb-b039-4321-ac58-fdcd3fe292a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b22f18cb-b039-4321-ac58-fdcd3fe292a7" (UID: "b22f18cb-b039-4321-ac58-fdcd3fe292a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:50:38.449980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.449891    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwdh8\" (UniqueName: \"kubernetes.io/projected/b22f18cb-b039-4321-ac58-fdcd3fe292a7-kube-api-access-bwdh8\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:50:38.449980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.449924    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b22f18cb-b039-4321-ac58-fdcd3fe292a7-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:50:38.449980 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:38.449939    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b22f18cb-b039-4321-ac58-fdcd3fe292a7-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:50:39.164907 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:39.164878    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"
Jul 28 01:50:39.165386 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:39.164886    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s" event={"ID":"b22f18cb-b039-4321-ac58-fdcd3fe292a7","Type":"ContainerDied","Data":"bd4847c4c7708e29e718ca703e0eefc2c2b210e5c6e9994a163073dadf699cc1"}
Jul 28 01:50:39.165386 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:39.165010    2578 scope.go:117] "RemoveContainer" containerID="e61ff11ac17888750bf589cb29d049c3f1635e65dd2ba7188dadbc03897a3595"
Jul 28 01:50:39.183042 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:39.183015    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"]
Jul 28 01:50:39.187488 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:39.187468    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-6bp7s"]
Jul 28 01:50:40.984998 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:50:40.984964    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b22f18cb-b039-4321-ac58-fdcd3fe292a7" path="/var/lib/kubelet/pods/b22f18cb-b039-4321-ac58-fdcd3fe292a7/volumes"
Jul 28 01:51:00.122541 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:00.122512    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc"
Jul 28 01:51:09.351098 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.351065    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"]
Jul 28 01:51:09.351568 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.351393    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b22f18cb-b039-4321-ac58-fdcd3fe292a7" containerName="main"
Jul 28 01:51:09.351568 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.351404    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b22f18cb-b039-4321-ac58-fdcd3fe292a7" containerName="main"
Jul 28 01:51:09.351568 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.351468    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b22f18cb-b039-4321-ac58-fdcd3fe292a7" containerName="main"
Jul 28 01:51:09.355690 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.355672    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.358450 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.358422    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 28 01:51:09.358577 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.358430    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-pvjf2\""
Jul 28 01:51:09.359957 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.359937    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 28 01:51:09.360069 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.359937    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 28 01:51:09.365238 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.365216    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"]
Jul 28 01:51:09.510855 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.510814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c5377bd-111a-47a6-a872-a3cf432c6174-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.511037 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.510867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8cw5\" (UniqueName: \"kubernetes.io/projected/8c5377bd-111a-47a6-a872-a3cf432c6174-kube-api-access-r8cw5\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.511037 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.510900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.511037 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.510917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.511037 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.510934    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.511177 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.511032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.611818 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.611728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.611818 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.611771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c5377bd-111a-47a6-a872-a3cf432c6174-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.611818 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.611814    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8cw5\" (UniqueName: \"kubernetes.io/projected/8c5377bd-111a-47a6-a872-a3cf432c6174-kube-api-access-r8cw5\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.612069 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.611852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.612069 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.611881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.612069 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.611907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.612223 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.612202    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.612362 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.612212    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.612362 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.612307    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.614103 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.614081    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.614392 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.614375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c5377bd-111a-47a6-a872-a3cf432c6174-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.622011 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.621986    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8cw5\" (UniqueName: \"kubernetes.io/projected/8c5377bd-111a-47a6-a872-a3cf432c6174-kube-api-access-r8cw5\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.666992 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.666963    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:09.820967 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:09.820931    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"]
Jul 28 01:51:09.823170 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:51:09.823142    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5377bd_111a_47a6_a872_a3cf432c6174.slice/crio-0c7062e18534cc6040d6b6f97280f0723682d8eca5c3f926a02c8bdc3e529fab WatchSource:0}: Error finding container 0c7062e18534cc6040d6b6f97280f0723682d8eca5c3f926a02c8bdc3e529fab: Status 404 returned error can't find the container with id 0c7062e18534cc6040d6b6f97280f0723682d8eca5c3f926a02c8bdc3e529fab
Jul 28 01:51:10.303729 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:10.303686    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg" event={"ID":"8c5377bd-111a-47a6-a872-a3cf432c6174","Type":"ContainerStarted","Data":"fd9825aeaecc3a837e761cbb7015ba10a5ea9b72a1be73683a152be0d203eb1e"}
Jul 28 01:51:10.303916 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:10.303740    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg" event={"ID":"8c5377bd-111a-47a6-a872-a3cf432c6174","Type":"ContainerStarted","Data":"0c7062e18534cc6040d6b6f97280f0723682d8eca5c3f926a02c8bdc3e529fab"}
Jul 28 01:51:10.324568 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:10.324502    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg" podStartSLOduration=1.324480579 podStartE2EDuration="1.324480579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:51:10.322905698 +0000 UTC m=+1513.900363328" watchObservedRunningTime="2026-07-28 01:51:10.324480579 +0000 UTC m=+1513.901938201"
Jul 28 01:51:10.667302 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:10.667218    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:19.667677 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:19.667593    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:19.680085 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:19.680057    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:20.360583 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:20.360553    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:43.961481 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:43.961445    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"]
Jul 28 01:51:43.971442 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:51:43.971406    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:51:43.971442 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:51:43.971442 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:51:43.971442 ip-10-0-139-81 kubenswrapper[2578]:  > containerID="fd9825aeaecc3a837e761cbb7015ba10a5ea9b72a1be73683a152be0d203eb1e" cmd=["/bin/sleep","15"]
Jul 28 01:51:43.971746 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:43.971459    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg" message="" err=<
Jul 28 01:51:43.971746 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:51:43.971746 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:51:43.971746 ip-10-0-139-81 kubenswrapper[2578]:  >
Jul 28 01:51:43.971746 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:51:43.971471    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:51:43.971746 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:51:43.971746 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:51:43.971746 ip-10-0-139-81 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg" podUID="8c5377bd-111a-47a6-a872-a3cf432c6174" containerName="main" containerID="cri-o://fd9825aeaecc3a837e761cbb7015ba10a5ea9b72a1be73683a152be0d203eb1e"
Jul 28 01:51:43.971746 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:43.971517    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg" podUID="8c5377bd-111a-47a6-a872-a3cf432c6174" containerName="main" containerID="cri-o://fd9825aeaecc3a837e761cbb7015ba10a5ea9b72a1be73683a152be0d203eb1e" gracePeriod=60
Jul 28 01:51:44.321988 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.321964    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:44.397623 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.397590    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-tmp-dir\") pod \"8c5377bd-111a-47a6-a872-a3cf432c6174\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") "
Jul 28 01:51:44.397816 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.397633    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-home\") pod \"8c5377bd-111a-47a6-a872-a3cf432c6174\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") "
Jul 28 01:51:44.397816 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.397662    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c5377bd-111a-47a6-a872-a3cf432c6174-tls-certs\") pod \"8c5377bd-111a-47a6-a872-a3cf432c6174\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") "
Jul 28 01:51:44.397816 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.397721    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-dshm\") pod \"8c5377bd-111a-47a6-a872-a3cf432c6174\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") "
Jul 28 01:51:44.397816 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.397775    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-model-cache\") pod \"8c5377bd-111a-47a6-a872-a3cf432c6174\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") "
Jul 28 01:51:44.397816 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.397801    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8cw5\" (UniqueName: \"kubernetes.io/projected/8c5377bd-111a-47a6-a872-a3cf432c6174-kube-api-access-r8cw5\") pod \"8c5377bd-111a-47a6-a872-a3cf432c6174\" (UID: \"8c5377bd-111a-47a6-a872-a3cf432c6174\") "
Jul 28 01:51:44.398058 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.397885    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c5377bd-111a-47a6-a872-a3cf432c6174" (UID: "8c5377bd-111a-47a6-a872-a3cf432c6174"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:51:44.398058 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.397898    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-home" (OuterVolumeSpecName: "home") pod "8c5377bd-111a-47a6-a872-a3cf432c6174" (UID: "8c5377bd-111a-47a6-a872-a3cf432c6174"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:51:44.398058 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.398053    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:51:44.398214 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.398071    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:51:44.398355 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.398305    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c5377bd-111a-47a6-a872-a3cf432c6174" (UID: "8c5377bd-111a-47a6-a872-a3cf432c6174"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:51:44.400576 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.400542    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c5377bd-111a-47a6-a872-a3cf432c6174-kube-api-access-r8cw5" (OuterVolumeSpecName: "kube-api-access-r8cw5") pod "8c5377bd-111a-47a6-a872-a3cf432c6174" (UID: "8c5377bd-111a-47a6-a872-a3cf432c6174"). InnerVolumeSpecName "kube-api-access-r8cw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:51:44.400691 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.400592    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-dshm" (OuterVolumeSpecName: "dshm") pod "8c5377bd-111a-47a6-a872-a3cf432c6174" (UID: "8c5377bd-111a-47a6-a872-a3cf432c6174"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:51:44.400691 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.400622    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c5377bd-111a-47a6-a872-a3cf432c6174-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c5377bd-111a-47a6-a872-a3cf432c6174" (UID: "8c5377bd-111a-47a6-a872-a3cf432c6174"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:51:44.456829 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.456801    2578 generic.go:356] "Generic (PLEG): container finished" podID="8c5377bd-111a-47a6-a872-a3cf432c6174" containerID="fd9825aeaecc3a837e761cbb7015ba10a5ea9b72a1be73683a152be0d203eb1e" exitCode=0
Jul 28 01:51:44.456961 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.456851    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"
Jul 28 01:51:44.456961 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.456850    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg" event={"ID":"8c5377bd-111a-47a6-a872-a3cf432c6174","Type":"ContainerDied","Data":"fd9825aeaecc3a837e761cbb7015ba10a5ea9b72a1be73683a152be0d203eb1e"}
Jul 28 01:51:44.457050 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.456972    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg" event={"ID":"8c5377bd-111a-47a6-a872-a3cf432c6174","Type":"ContainerDied","Data":"0c7062e18534cc6040d6b6f97280f0723682d8eca5c3f926a02c8bdc3e529fab"}
Jul 28 01:51:44.457050 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.456996    2578 scope.go:117] "RemoveContainer" containerID="fd9825aeaecc3a837e761cbb7015ba10a5ea9b72a1be73683a152be0d203eb1e"
Jul 28 01:51:44.465683 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.465662    2578 scope.go:117] "RemoveContainer" containerID="fd9825aeaecc3a837e761cbb7015ba10a5ea9b72a1be73683a152be0d203eb1e"
Jul 28 01:51:44.465938 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:51:44.465917    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9825aeaecc3a837e761cbb7015ba10a5ea9b72a1be73683a152be0d203eb1e\": container with ID starting with fd9825aeaecc3a837e761cbb7015ba10a5ea9b72a1be73683a152be0d203eb1e not found: ID does not exist" containerID="fd9825aeaecc3a837e761cbb7015ba10a5ea9b72a1be73683a152be0d203eb1e"
Jul 28 01:51:44.466024 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.465945    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9825aeaecc3a837e761cbb7015ba10a5ea9b72a1be73683a152be0d203eb1e"} err="failed to get container status \"fd9825aeaecc3a837e761cbb7015ba10a5ea9b72a1be73683a152be0d203eb1e\": rpc error: code = NotFound desc = could not find container \"fd9825aeaecc3a837e761cbb7015ba10a5ea9b72a1be73683a152be0d203eb1e\": container with ID starting with fd9825aeaecc3a837e761cbb7015ba10a5ea9b72a1be73683a152be0d203eb1e not found: ID does not exist"
Jul 28 01:51:44.478547 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.478525    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"]
Jul 28 01:51:44.484378 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.484355    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-jxwbg"]
Jul 28 01:51:44.499298 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.499254    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:51:44.499298 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.499272    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c5377bd-111a-47a6-a872-a3cf432c6174-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:51:44.499298 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.499281    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8cw5\" (UniqueName: \"kubernetes.io/projected/8c5377bd-111a-47a6-a872-a3cf432c6174-kube-api-access-r8cw5\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:51:44.499298 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.499290    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c5377bd-111a-47a6-a872-a3cf432c6174-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:51:44.984716 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:51:44.984684    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c5377bd-111a-47a6-a872-a3cf432c6174" path="/var/lib/kubelet/pods/8c5377bd-111a-47a6-a872-a3cf432c6174/volumes"
Jul 28 01:52:11.269457 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.269425    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"]
Jul 28 01:52:11.269876 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.269730    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c5377bd-111a-47a6-a872-a3cf432c6174" containerName="main"
Jul 28 01:52:11.269876 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.269741    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c5377bd-111a-47a6-a872-a3cf432c6174" containerName="main"
Jul 28 01:52:11.269876 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.269798    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c5377bd-111a-47a6-a872-a3cf432c6174" containerName="main"
Jul 28 01:52:11.273981 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.273959    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.277844 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.277819    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-t564d\""
Jul 28 01:52:11.278594 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.278574    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 28 01:52:11.278747 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.278711    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 28 01:52:11.279005 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.278988    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-llc24\""
Jul 28 01:52:11.280367 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.280349    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 28 01:52:11.288122 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.288100    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"]
Jul 28 01:52:11.414534 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.414502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.414709 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.414545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.414784 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.414573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9czv\" (UniqueName: \"kubernetes.io/projected/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-kube-api-access-g9czv\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.415196 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.415110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.415621 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.415235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.415621 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.415274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.516119 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.516086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.516304 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.516128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.516304 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.516157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.516304 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.516180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.516304 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.516206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9czv\" (UniqueName: \"kubernetes.io/projected/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-kube-api-access-g9czv\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.516304 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.516242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.516624 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.516594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.516755 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.516633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.516755 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.516656    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.518585 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.518552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.518719 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.518694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.527372 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.527301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9czv\" (UniqueName: \"kubernetes.io/projected/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-kube-api-access-g9czv\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.586141 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.586103    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:11.738916 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:11.738892    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"]
Jul 28 01:52:11.741177 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:52:11.741145    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b8d990_a4d6_40bd_8ce9_97b23ed1e2ec.slice/crio-1f678357f89462873110ec2ac15d572c2018407782283284a32fa58fb9bdd88c WatchSource:0}: Error finding container 1f678357f89462873110ec2ac15d572c2018407782283284a32fa58fb9bdd88c: Status 404 returned error can't find the container with id 1f678357f89462873110ec2ac15d572c2018407782283284a32fa58fb9bdd88c
Jul 28 01:52:12.573211 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:12.573169    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k" event={"ID":"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec","Type":"ContainerStarted","Data":"1f678357f89462873110ec2ac15d572c2018407782283284a32fa58fb9bdd88c"}
Jul 28 01:52:13.580124 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:13.580088    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k" event={"ID":"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec","Type":"ContainerStarted","Data":"9fde138deb3a4b2367037951c663d3ef5c1ebea2499e0f98d8b5a72c5980e167"}
Jul 28 01:52:13.580591 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:13.580233    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:14.586821 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:14.586781    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k" event={"ID":"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec","Type":"ContainerStarted","Data":"ad9cc500678fb538c635a3dcda83db897bc46d936ec4aec1ad9b83a7e9b5487b"}
Jul 28 01:52:14.606675 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:14.606621    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k" podStartSLOduration=2.246768512 podStartE2EDuration="3.606604373s" totalImagesPullingTime="1.359835861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:52:14.605513209 +0000 UTC m=+1578.182970849" watchObservedRunningTime="2026-07-28 01:52:14.606604373 +0000 UTC m=+1578.184061989"
Jul 28 01:52:15.587030 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:15.586994    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:21.586510 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:21.586478    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:21.598700 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:21.598674    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:21.629211 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:21.629183    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:32.634150 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:32.634120    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:45.270391 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.270354    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"]
Jul 28 01:52:45.280681 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:52:45.280644    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:52:45.280681 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:52:45.280681 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:52:45.280681 ip-10-0-139-81 kubenswrapper[2578]:  > containerID="ad9cc500678fb538c635a3dcda83db897bc46d936ec4aec1ad9b83a7e9b5487b" cmd=["/bin/sleep","15"]
Jul 28 01:52:45.280960 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.280701    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k" message="" err=<
Jul 28 01:52:45.280960 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:52:45.280960 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:52:45.280960 ip-10-0-139-81 kubenswrapper[2578]:  >
Jul 28 01:52:45.280960 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:52:45.280715    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:52:45.280960 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:52:45.280960 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:52:45.280960 ip-10-0-139-81 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k" podUID="49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec" containerName="main" containerID="cri-o://ad9cc500678fb538c635a3dcda83db897bc46d936ec4aec1ad9b83a7e9b5487b"
Jul 28 01:52:45.280960 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.280771    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k" podUID="49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec" containerName="main" containerID="cri-o://ad9cc500678fb538c635a3dcda83db897bc46d936ec4aec1ad9b83a7e9b5487b" gracePeriod=60
Jul 28 01:52:45.498681 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.498653    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k" podUID="49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec" containerName="llm-d-routing-sidecar" containerID="cri-o://9fde138deb3a4b2367037951c663d3ef5c1ebea2499e0f98d8b5a72c5980e167" gracePeriod=60
Jul 28 01:52:45.619183 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.619148    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k" podUID="49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8001/health\": dial tcp 10.134.0.35:8001: connect: connection refused"
Jul 28 01:52:45.720341 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.720296    2578 generic.go:356] "Generic (PLEG): container finished" podID="49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec" containerID="ad9cc500678fb538c635a3dcda83db897bc46d936ec4aec1ad9b83a7e9b5487b" exitCode=0
Jul 28 01:52:45.720341 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.720320    2578 generic.go:356] "Generic (PLEG): container finished" podID="49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec" containerID="9fde138deb3a4b2367037951c663d3ef5c1ebea2499e0f98d8b5a72c5980e167" exitCode=0
Jul 28 01:52:45.720527 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.720376    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k" event={"ID":"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec","Type":"ContainerDied","Data":"ad9cc500678fb538c635a3dcda83db897bc46d936ec4aec1ad9b83a7e9b5487b"}
Jul 28 01:52:45.720527 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.720445    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k" event={"ID":"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec","Type":"ContainerDied","Data":"9fde138deb3a4b2367037951c663d3ef5c1ebea2499e0f98d8b5a72c5980e167"}
Jul 28 01:52:45.744554 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.744533    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:45.788612 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.788587    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-home\") pod \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") "
Jul 28 01:52:45.788612 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.788613    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-tmp-dir\") pod \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") "
Jul 28 01:52:45.788804 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.788631    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-tls-certs\") pod \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") "
Jul 28 01:52:45.788804 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.788684    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-dshm\") pod \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") "
Jul 28 01:52:45.788804 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.788709    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9czv\" (UniqueName: \"kubernetes.io/projected/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-kube-api-access-g9czv\") pod \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") "
Jul 28 01:52:45.788804 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.788755    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-model-cache\") pod \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\" (UID: \"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec\") "
Jul 28 01:52:45.788984 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.788860    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec" (UID: "49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:52:45.788984 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.788874    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-home" (OuterVolumeSpecName: "home") pod "49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec" (UID: "49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:52:45.789079 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.788983    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:52:45.789079 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.789065    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:52:45.789079 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.789063    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec" (UID: "49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:52:45.790848 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.790824    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec" (UID: "49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:52:45.790925 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.790870    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-kube-api-access-g9czv" (OuterVolumeSpecName: "kube-api-access-g9czv") pod "49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec" (UID: "49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec"). InnerVolumeSpecName "kube-api-access-g9czv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:52:45.790925 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.790873    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-dshm" (OuterVolumeSpecName: "dshm") pod "49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec" (UID: "49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:52:45.889432 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.889372    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:52:45.889432 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.889398    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9czv\" (UniqueName: \"kubernetes.io/projected/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-kube-api-access-g9czv\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:52:45.889432 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.889409    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:52:45.889432 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:45.889418    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:52:46.726001 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:46.725968    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"
Jul 28 01:52:46.726430 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:46.725968    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k" event={"ID":"49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec","Type":"ContainerDied","Data":"1f678357f89462873110ec2ac15d572c2018407782283284a32fa58fb9bdd88c"}
Jul 28 01:52:46.726430 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:46.726113    2578 scope.go:117] "RemoveContainer" containerID="ad9cc500678fb538c635a3dcda83db897bc46d936ec4aec1ad9b83a7e9b5487b"
Jul 28 01:52:46.735101 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:46.735080    2578 scope.go:117] "RemoveContainer" containerID="9fde138deb3a4b2367037951c663d3ef5c1ebea2499e0f98d8b5a72c5980e167"
Jul 28 01:52:46.747144 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:46.747124    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"]
Jul 28 01:52:46.751840 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:46.751819    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-78b8k"]
Jul 28 01:52:46.986004 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:52:46.985929    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec" path="/var/lib/kubelet/pods/49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec/volumes"
Jul 28 01:53:28.955099 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:28.955063    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"]
Jul 28 01:53:28.955606 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:28.955398    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec" containerName="llm-d-routing-sidecar"
Jul 28 01:53:28.955606 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:28.955411    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec" containerName="llm-d-routing-sidecar"
Jul 28 01:53:28.955606 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:28.955451    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec" containerName="main"
Jul 28 01:53:28.955606 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:28.955457    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec" containerName="main"
Jul 28 01:53:28.955606 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:28.955509    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec" containerName="main"
Jul 28 01:53:28.955606 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:28.955525    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b8d990-a4d6-40bd-8ce9-97b23ed1e2ec" containerName="llm-d-routing-sidecar"
Jul 28 01:53:28.958526 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:28.958506    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:28.961590 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:28.961564    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 28 01:53:28.961732 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:28.961574    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 28 01:53:28.962635 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:28.962613    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-5nqpm\""
Jul 28 01:53:28.962635 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:28.962631    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Jul 28 01:53:28.962789 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:28.962700    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 28 01:53:28.971714 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:28.971693    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"]
Jul 28 01:53:29.037106 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.037068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.037106 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.037113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-home\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.037372 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.037133    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-model-cache\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.037372 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.037165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-dshm\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.037372 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.037242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95d47f37-2be0-44f9-a52b-5e3df1db42bf-tls-certs\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.037372 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.037281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.037372 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.037352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd9fr\" (UniqueName: \"kubernetes.io/projected/95d47f37-2be0-44f9-a52b-5e3df1db42bf-kube-api-access-rd9fr\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.138176 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.138136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95d47f37-2be0-44f9-a52b-5e3df1db42bf-tls-certs\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.138176 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.138188    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.138474 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.138217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd9fr\" (UniqueName: \"kubernetes.io/projected/95d47f37-2be0-44f9-a52b-5e3df1db42bf-kube-api-access-rd9fr\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.138474 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.138245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.138474 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.138272    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-home\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.138474 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.138293    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-model-cache\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.138474 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.138318    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-dshm\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.138720 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.138692    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.138786 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.138728    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.138844 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.138776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-home\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.138899 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.138840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-model-cache\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.140755 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.140736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-dshm\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.141040 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.141018    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95d47f37-2be0-44f9-a52b-5e3df1db42bf-tls-certs\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.148487 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.148467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd9fr\" (UniqueName: \"kubernetes.io/projected/95d47f37-2be0-44f9-a52b-5e3df1db42bf-kube-api-access-rd9fr\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.269491 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.269459    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:29.407598 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.407571    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"]
Jul 28 01:53:29.409735 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:53:29.409706    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d47f37_2be0_44f9_a52b_5e3df1db42bf.slice/crio-7c622f048e0cf04ac7bcad5c407c8577fd5d01e719dc5e94e002ec6e2d4e766e WatchSource:0}: Error finding container 7c622f048e0cf04ac7bcad5c407c8577fd5d01e719dc5e94e002ec6e2d4e766e: Status 404 returned error can't find the container with id 7c622f048e0cf04ac7bcad5c407c8577fd5d01e719dc5e94e002ec6e2d4e766e
Jul 28 01:53:29.902928 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.902894    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz" event={"ID":"95d47f37-2be0-44f9-a52b-5e3df1db42bf","Type":"ContainerStarted","Data":"608613854fb4dd65e807eb2ea35c2d65d76f873f6b5808cf1f44668b6709f14a"}
Jul 28 01:53:29.903108 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:29.902945    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz" event={"ID":"95d47f37-2be0-44f9-a52b-5e3df1db42bf","Type":"ContainerStarted","Data":"7c622f048e0cf04ac7bcad5c407c8577fd5d01e719dc5e94e002ec6e2d4e766e"}
Jul 28 01:53:34.928553 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:34.928519    2578 generic.go:356] "Generic (PLEG): container finished" podID="95d47f37-2be0-44f9-a52b-5e3df1db42bf" containerID="608613854fb4dd65e807eb2ea35c2d65d76f873f6b5808cf1f44668b6709f14a" exitCode=0
Jul 28 01:53:34.928998 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:34.928571    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz" event={"ID":"95d47f37-2be0-44f9-a52b-5e3df1db42bf","Type":"ContainerDied","Data":"608613854fb4dd65e807eb2ea35c2d65d76f873f6b5808cf1f44668b6709f14a"}
Jul 28 01:53:35.934873 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:35.934837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz" event={"ID":"95d47f37-2be0-44f9-a52b-5e3df1db42bf","Type":"ContainerStarted","Data":"d48ca5d747fe365aca9cb9f19dc957d3e3a8885533b1752b2bdf5e9aeec1e3e9"}
Jul 28 01:53:35.955412 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:35.955361    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz" podStartSLOduration=3.07413117 podStartE2EDuration="7.955344987s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.881213817s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:53:35.954749496 +0000 UTC m=+1659.532207114" watchObservedRunningTime="2026-07-28 01:53:35.955344987 +0000 UTC m=+1659.532802621"
Jul 28 01:53:36.270301 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:36.270269    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:39.269680 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:39.269637    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:39.282381 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:39.282354    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:53:39.963586 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:53:39.963555    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:54:41.753248 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:41.753166    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"]
Jul 28 01:54:41.762388 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:54:41.762347    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:54:41.762388 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:54:41.762388 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:54:41.762388 ip-10-0-139-81 kubenswrapper[2578]:  > containerID="d48ca5d747fe365aca9cb9f19dc957d3e3a8885533b1752b2bdf5e9aeec1e3e9" cmd=["/bin/sleep","15"]
Jul 28 01:54:41.762709 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:41.762404    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz" message="" err=<
Jul 28 01:54:41.762709 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:54:41.762709 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:54:41.762709 ip-10-0-139-81 kubenswrapper[2578]:  >
Jul 28 01:54:41.762709 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:54:41.762416    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:54:41.762709 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:54:41.762709 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:54:41.762709 ip-10-0-139-81 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz" podUID="95d47f37-2be0-44f9-a52b-5e3df1db42bf" containerName="main" containerID="cri-o://d48ca5d747fe365aca9cb9f19dc957d3e3a8885533b1752b2bdf5e9aeec1e3e9"
Jul 28 01:54:41.762709 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:41.762463    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz" podUID="95d47f37-2be0-44f9-a52b-5e3df1db42bf" containerName="main" containerID="cri-o://d48ca5d747fe365aca9cb9f19dc957d3e3a8885533b1752b2bdf5e9aeec1e3e9" gracePeriod=60
Jul 28 01:54:41.953200 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:41.953159    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz" podUID="95d47f37-2be0-44f9-a52b-5e3df1db42bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Jul 28 01:54:42.119446 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.119419    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:54:42.141373 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.140871    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-kserve-provision-location\") pod \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") "
Jul 28 01:54:42.141373 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.140936    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-home\") pod \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") "
Jul 28 01:54:42.141373 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.140967    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95d47f37-2be0-44f9-a52b-5e3df1db42bf-tls-certs\") pod \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") "
Jul 28 01:54:42.141373 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.141005    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-tmp-dir\") pod \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") "
Jul 28 01:54:42.141373 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.141051    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-dshm\") pod \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") "
Jul 28 01:54:42.141373 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.141094    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-model-cache\") pod \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") "
Jul 28 01:54:42.141373 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.141132    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd9fr\" (UniqueName: \"kubernetes.io/projected/95d47f37-2be0-44f9-a52b-5e3df1db42bf-kube-api-access-rd9fr\") pod \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\" (UID: \"95d47f37-2be0-44f9-a52b-5e3df1db42bf\") "
Jul 28 01:54:42.144277 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.144233    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d47f37-2be0-44f9-a52b-5e3df1db42bf-kube-api-access-rd9fr" (OuterVolumeSpecName: "kube-api-access-rd9fr") pod "95d47f37-2be0-44f9-a52b-5e3df1db42bf" (UID: "95d47f37-2be0-44f9-a52b-5e3df1db42bf"). InnerVolumeSpecName "kube-api-access-rd9fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:54:42.144578 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.144513    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95d47f37-2be0-44f9-a52b-5e3df1db42bf" (UID: "95d47f37-2be0-44f9-a52b-5e3df1db42bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:54:42.145153 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.145011    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "95d47f37-2be0-44f9-a52b-5e3df1db42bf" (UID: "95d47f37-2be0-44f9-a52b-5e3df1db42bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:54:42.145463 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.145427    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-home" (OuterVolumeSpecName: "home") pod "95d47f37-2be0-44f9-a52b-5e3df1db42bf" (UID: "95d47f37-2be0-44f9-a52b-5e3df1db42bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:54:42.164472 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.164396    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d47f37-2be0-44f9-a52b-5e3df1db42bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95d47f37-2be0-44f9-a52b-5e3df1db42bf" (UID: "95d47f37-2be0-44f9-a52b-5e3df1db42bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:54:42.164618 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.164404    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-dshm" (OuterVolumeSpecName: "dshm") pod "95d47f37-2be0-44f9-a52b-5e3df1db42bf" (UID: "95d47f37-2be0-44f9-a52b-5e3df1db42bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:54:42.214285 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.214243    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "95d47f37-2be0-44f9-a52b-5e3df1db42bf" (UID: "95d47f37-2be0-44f9-a52b-5e3df1db42bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:54:42.223782 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.223743    2578 generic.go:356] "Generic (PLEG): container finished" podID="95d47f37-2be0-44f9-a52b-5e3df1db42bf" containerID="d48ca5d747fe365aca9cb9f19dc957d3e3a8885533b1752b2bdf5e9aeec1e3e9" exitCode=0
Jul 28 01:54:42.223963 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.223776    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz" event={"ID":"95d47f37-2be0-44f9-a52b-5e3df1db42bf","Type":"ContainerDied","Data":"d48ca5d747fe365aca9cb9f19dc957d3e3a8885533b1752b2bdf5e9aeec1e3e9"}
Jul 28 01:54:42.223963 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.223821    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"
Jul 28 01:54:42.223963 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.223876    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz" event={"ID":"95d47f37-2be0-44f9-a52b-5e3df1db42bf","Type":"ContainerDied","Data":"7c622f048e0cf04ac7bcad5c407c8577fd5d01e719dc5e94e002ec6e2d4e766e"}
Jul 28 01:54:42.223963 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.223906    2578 scope.go:117] "RemoveContainer" containerID="d48ca5d747fe365aca9cb9f19dc957d3e3a8885533b1752b2bdf5e9aeec1e3e9"
Jul 28 01:54:42.233903 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.233883    2578 scope.go:117] "RemoveContainer" containerID="608613854fb4dd65e807eb2ea35c2d65d76f873f6b5808cf1f44668b6709f14a"
Jul 28 01:54:42.242071 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.242044    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:54:42.242071 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.242066    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd9fr\" (UniqueName: \"kubernetes.io/projected/95d47f37-2be0-44f9-a52b-5e3df1db42bf-kube-api-access-rd9fr\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:54:42.242269 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.242076    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-kserve-provision-location\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:54:42.242269 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.242087    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:54:42.242269 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.242096    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95d47f37-2be0-44f9-a52b-5e3df1db42bf-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:54:42.242269 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.242127    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:54:42.242269 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.242135    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d47f37-2be0-44f9-a52b-5e3df1db42bf-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:54:42.242933 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.242916    2578 scope.go:117] "RemoveContainer" containerID="d48ca5d747fe365aca9cb9f19dc957d3e3a8885533b1752b2bdf5e9aeec1e3e9"
Jul 28 01:54:42.243210 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:54:42.243187    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d48ca5d747fe365aca9cb9f19dc957d3e3a8885533b1752b2bdf5e9aeec1e3e9\": container with ID starting with d48ca5d747fe365aca9cb9f19dc957d3e3a8885533b1752b2bdf5e9aeec1e3e9 not found: ID does not exist" containerID="d48ca5d747fe365aca9cb9f19dc957d3e3a8885533b1752b2bdf5e9aeec1e3e9"
Jul 28 01:54:42.243286 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.243222    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48ca5d747fe365aca9cb9f19dc957d3e3a8885533b1752b2bdf5e9aeec1e3e9"} err="failed to get container status \"d48ca5d747fe365aca9cb9f19dc957d3e3a8885533b1752b2bdf5e9aeec1e3e9\": rpc error: code = NotFound desc = could not find container \"d48ca5d747fe365aca9cb9f19dc957d3e3a8885533b1752b2bdf5e9aeec1e3e9\": container with ID starting with d48ca5d747fe365aca9cb9f19dc957d3e3a8885533b1752b2bdf5e9aeec1e3e9 not found: ID does not exist"
Jul 28 01:54:42.243286 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.243247    2578 scope.go:117] "RemoveContainer" containerID="608613854fb4dd65e807eb2ea35c2d65d76f873f6b5808cf1f44668b6709f14a"
Jul 28 01:54:42.243539 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:54:42.243514    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608613854fb4dd65e807eb2ea35c2d65d76f873f6b5808cf1f44668b6709f14a\": container with ID starting with 608613854fb4dd65e807eb2ea35c2d65d76f873f6b5808cf1f44668b6709f14a not found: ID does not exist" containerID="608613854fb4dd65e807eb2ea35c2d65d76f873f6b5808cf1f44668b6709f14a"
Jul 28 01:54:42.243584 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.243546    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608613854fb4dd65e807eb2ea35c2d65d76f873f6b5808cf1f44668b6709f14a"} err="failed to get container status \"608613854fb4dd65e807eb2ea35c2d65d76f873f6b5808cf1f44668b6709f14a\": rpc error: code = NotFound desc = could not find container \"608613854fb4dd65e807eb2ea35c2d65d76f873f6b5808cf1f44668b6709f14a\": container with ID starting with 608613854fb4dd65e807eb2ea35c2d65d76f873f6b5808cf1f44668b6709f14a not found: ID does not exist"
Jul 28 01:54:42.247648 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.247624    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"]
Jul 28 01:54:42.251977 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.251956    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-rhbbz"]
Jul 28 01:54:42.985006 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:54:42.984975    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95d47f37-2be0-44f9-a52b-5e3df1db42bf" path="/var/lib/kubelet/pods/95d47f37-2be0-44f9-a52b-5e3df1db42bf/volumes"
Jul 28 01:55:09.758762 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.758723    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"]
Jul 28 01:55:09.759132 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.759047    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95d47f37-2be0-44f9-a52b-5e3df1db42bf" containerName="storage-initializer"
Jul 28 01:55:09.759132 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.759059    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="95d47f37-2be0-44f9-a52b-5e3df1db42bf" containerName="storage-initializer"
Jul 28 01:55:09.759132 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.759067    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95d47f37-2be0-44f9-a52b-5e3df1db42bf" containerName="main"
Jul 28 01:55:09.759132 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.759073    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="95d47f37-2be0-44f9-a52b-5e3df1db42bf" containerName="main"
Jul 28 01:55:09.759267 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.759135    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="95d47f37-2be0-44f9-a52b-5e3df1db42bf" containerName="main"
Jul 28 01:55:09.761236 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.761215    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.764121 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.764094    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 28 01:55:09.765501 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.765480    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 28 01:55:09.765615 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.765519    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-kqnzw\""
Jul 28 01:55:09.765615 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.765579    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 28 01:55:09.765615 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.765597    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 28 01:55:09.776620 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.776595    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"]
Jul 28 01:55:09.837464 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.837432    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.837696 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.837469    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.837696 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.837497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a250a9-58f2-4a12-8105-f546b8a85a02-tls-certs\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.837696 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.837546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6gzk\" (UniqueName: \"kubernetes.io/projected/84a250a9-58f2-4a12-8105-f546b8a85a02-kube-api-access-m6gzk\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.837696 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.837577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-home\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.837696 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.837625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-dshm\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.837696 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.837646    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-model-cache\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.938727 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.938691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a250a9-58f2-4a12-8105-f546b8a85a02-tls-certs\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.938945 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.938742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6gzk\" (UniqueName: \"kubernetes.io/projected/84a250a9-58f2-4a12-8105-f546b8a85a02-kube-api-access-m6gzk\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.938945 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.938911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-home\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.939074 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.938980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-dshm\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.939074 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.939016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-model-cache\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.939183 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.939063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.939183 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.939117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.939377 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.939354    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-home\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.939523 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.939498    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.939523 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.939392    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.939523 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.939431    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-model-cache\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.941143 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.941124    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-dshm\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.941241 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.941215    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a250a9-58f2-4a12-8105-f546b8a85a02-tls-certs\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:09.946797 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:09.946778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6gzk\" (UniqueName: \"kubernetes.io/projected/84a250a9-58f2-4a12-8105-f546b8a85a02-kube-api-access-m6gzk\") pod \"tokenizer-migration-path-test-kserve-569c688787-kzwmm\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:10.071563 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:10.071518    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:10.210623 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:10.210595    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"]
Jul 28 01:55:10.213183 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:55:10.213151    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a250a9_58f2_4a12_8105_f546b8a85a02.slice/crio-6e9db473ed56e81b38d59fdaf140a01edf4fe419bfee8e9bcbc7c847106dbf96 WatchSource:0}: Error finding container 6e9db473ed56e81b38d59fdaf140a01edf4fe419bfee8e9bcbc7c847106dbf96: Status 404 returned error can't find the container with id 6e9db473ed56e81b38d59fdaf140a01edf4fe419bfee8e9bcbc7c847106dbf96
Jul 28 01:55:10.343996 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:10.343897    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm" event={"ID":"84a250a9-58f2-4a12-8105-f546b8a85a02","Type":"ContainerStarted","Data":"2dc04364a5f28f670f270f3c8777628504bc3473e743acd4c396b05ca4913c54"}
Jul 28 01:55:10.344181 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:10.343995    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm" event={"ID":"84a250a9-58f2-4a12-8105-f546b8a85a02","Type":"ContainerStarted","Data":"6e9db473ed56e81b38d59fdaf140a01edf4fe419bfee8e9bcbc7c847106dbf96"}
Jul 28 01:55:15.368526 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:15.368487    2578 generic.go:356] "Generic (PLEG): container finished" podID="84a250a9-58f2-4a12-8105-f546b8a85a02" containerID="2dc04364a5f28f670f270f3c8777628504bc3473e743acd4c396b05ca4913c54" exitCode=0
Jul 28 01:55:15.368979 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:15.368561    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm" event={"ID":"84a250a9-58f2-4a12-8105-f546b8a85a02","Type":"ContainerDied","Data":"2dc04364a5f28f670f270f3c8777628504bc3473e743acd4c396b05ca4913c54"}
Jul 28 01:55:16.374643 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:16.374599    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm" event={"ID":"84a250a9-58f2-4a12-8105-f546b8a85a02","Type":"ContainerStarted","Data":"7606771231e2ead4521024517c950b0d87e7979a6f2f9f0979e4e053ff180daa"}
Jul 28 01:55:16.396364 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:16.396296    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm" podStartSLOduration=2.803931273 podStartE2EDuration="7.396280296s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.592349023s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:55:16.394557936 +0000 UTC m=+1759.972015589" watchObservedRunningTime="2026-07-28 01:55:16.396280296 +0000 UTC m=+1759.973737913"
Jul 28 01:55:17.072640 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:17.072607    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:20.072488 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:20.072455    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:20.084931 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:20.084897    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:20.402535 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:20.402447    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:48.795612 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:48.795526    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"]
Jul 28 01:55:48.804625 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:55:48.804590    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:55:48.804625 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:55:48.804625 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:55:48.804625 ip-10-0-139-81 kubenswrapper[2578]:  > containerID="7606771231e2ead4521024517c950b0d87e7979a6f2f9f0979e4e053ff180daa" cmd=["/bin/sleep","15"]
Jul 28 01:55:48.804880 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:48.804635    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm" message="" err=<
Jul 28 01:55:48.804880 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:55:48.804880 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:55:48.804880 ip-10-0-139-81 kubenswrapper[2578]:  >
Jul 28 01:55:48.804880 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:55:48.804647    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:55:48.804880 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:55:48.804880 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:55:48.804880 ip-10-0-139-81 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm" podUID="84a250a9-58f2-4a12-8105-f546b8a85a02" containerName="main" containerID="cri-o://7606771231e2ead4521024517c950b0d87e7979a6f2f9f0979e4e053ff180daa"
Jul 28 01:55:48.804880 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:48.804680    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm" podUID="84a250a9-58f2-4a12-8105-f546b8a85a02" containerName="main" containerID="cri-o://7606771231e2ead4521024517c950b0d87e7979a6f2f9f0979e4e053ff180daa" gracePeriod=60
Jul 28 01:55:49.158348 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.158297    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:49.276215 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.276182    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-kserve-provision-location\") pod \"84a250a9-58f2-4a12-8105-f546b8a85a02\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") "
Jul 28 01:55:49.276426 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.276232    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-tmp-dir\") pod \"84a250a9-58f2-4a12-8105-f546b8a85a02\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") "
Jul 28 01:55:49.276426 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.276270    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a250a9-58f2-4a12-8105-f546b8a85a02-tls-certs\") pod \"84a250a9-58f2-4a12-8105-f546b8a85a02\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") "
Jul 28 01:55:49.276426 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.276299    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-dshm\") pod \"84a250a9-58f2-4a12-8105-f546b8a85a02\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") "
Jul 28 01:55:49.276426 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.276317    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-home\") pod \"84a250a9-58f2-4a12-8105-f546b8a85a02\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") "
Jul 28 01:55:49.276426 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.276376    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6gzk\" (UniqueName: \"kubernetes.io/projected/84a250a9-58f2-4a12-8105-f546b8a85a02-kube-api-access-m6gzk\") pod \"84a250a9-58f2-4a12-8105-f546b8a85a02\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") "
Jul 28 01:55:49.276712 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.276451    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-model-cache\") pod \"84a250a9-58f2-4a12-8105-f546b8a85a02\" (UID: \"84a250a9-58f2-4a12-8105-f546b8a85a02\") "
Jul 28 01:55:49.276712 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.276562    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "84a250a9-58f2-4a12-8105-f546b8a85a02" (UID: "84a250a9-58f2-4a12-8105-f546b8a85a02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:55:49.276712 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.276631    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-home" (OuterVolumeSpecName: "home") pod "84a250a9-58f2-4a12-8105-f546b8a85a02" (UID: "84a250a9-58f2-4a12-8105-f546b8a85a02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:55:49.276820 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.276755    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:55:49.276820 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.276775    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:55:49.276820 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.276794    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-model-cache" (OuterVolumeSpecName: "model-cache") pod "84a250a9-58f2-4a12-8105-f546b8a85a02" (UID: "84a250a9-58f2-4a12-8105-f546b8a85a02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:55:49.278667 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.278590    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-dshm" (OuterVolumeSpecName: "dshm") pod "84a250a9-58f2-4a12-8105-f546b8a85a02" (UID: "84a250a9-58f2-4a12-8105-f546b8a85a02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:55:49.278797 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.278687    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a250a9-58f2-4a12-8105-f546b8a85a02-kube-api-access-m6gzk" (OuterVolumeSpecName: "kube-api-access-m6gzk") pod "84a250a9-58f2-4a12-8105-f546b8a85a02" (UID: "84a250a9-58f2-4a12-8105-f546b8a85a02"). InnerVolumeSpecName "kube-api-access-m6gzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:55:49.278797 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.278707    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a250a9-58f2-4a12-8105-f546b8a85a02-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84a250a9-58f2-4a12-8105-f546b8a85a02" (UID: "84a250a9-58f2-4a12-8105-f546b8a85a02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:55:49.337146 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.337040    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "84a250a9-58f2-4a12-8105-f546b8a85a02" (UID: "84a250a9-58f2-4a12-8105-f546b8a85a02"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:55:49.377357 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.377288    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:55:49.377357 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.377325    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6gzk\" (UniqueName: \"kubernetes.io/projected/84a250a9-58f2-4a12-8105-f546b8a85a02-kube-api-access-m6gzk\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:55:49.377569 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.377368    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:55:49.377569 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.377381    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84a250a9-58f2-4a12-8105-f546b8a85a02-kserve-provision-location\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:55:49.377569 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.377394    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84a250a9-58f2-4a12-8105-f546b8a85a02-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:55:49.517040 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.517002    2578 generic.go:356] "Generic (PLEG): container finished" podID="84a250a9-58f2-4a12-8105-f546b8a85a02" containerID="7606771231e2ead4521024517c950b0d87e7979a6f2f9f0979e4e053ff180daa" exitCode=0
Jul 28 01:55:49.517224 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.517052    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"
Jul 28 01:55:49.517224 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.517084    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm" event={"ID":"84a250a9-58f2-4a12-8105-f546b8a85a02","Type":"ContainerDied","Data":"7606771231e2ead4521024517c950b0d87e7979a6f2f9f0979e4e053ff180daa"}
Jul 28 01:55:49.517224 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.517139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm" event={"ID":"84a250a9-58f2-4a12-8105-f546b8a85a02","Type":"ContainerDied","Data":"6e9db473ed56e81b38d59fdaf140a01edf4fe419bfee8e9bcbc7c847106dbf96"}
Jul 28 01:55:49.517224 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.517163    2578 scope.go:117] "RemoveContainer" containerID="7606771231e2ead4521024517c950b0d87e7979a6f2f9f0979e4e053ff180daa"
Jul 28 01:55:49.528133 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.528116    2578 scope.go:117] "RemoveContainer" containerID="2dc04364a5f28f670f270f3c8777628504bc3473e743acd4c396b05ca4913c54"
Jul 28 01:55:49.536450 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.536432    2578 scope.go:117] "RemoveContainer" containerID="7606771231e2ead4521024517c950b0d87e7979a6f2f9f0979e4e053ff180daa"
Jul 28 01:55:49.536750 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:55:49.536728    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7606771231e2ead4521024517c950b0d87e7979a6f2f9f0979e4e053ff180daa\": container with ID starting with 7606771231e2ead4521024517c950b0d87e7979a6f2f9f0979e4e053ff180daa not found: ID does not exist" containerID="7606771231e2ead4521024517c950b0d87e7979a6f2f9f0979e4e053ff180daa"
Jul 28 01:55:49.536799 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.536758    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7606771231e2ead4521024517c950b0d87e7979a6f2f9f0979e4e053ff180daa"} err="failed to get container status \"7606771231e2ead4521024517c950b0d87e7979a6f2f9f0979e4e053ff180daa\": rpc error: code = NotFound desc = could not find container \"7606771231e2ead4521024517c950b0d87e7979a6f2f9f0979e4e053ff180daa\": container with ID starting with 7606771231e2ead4521024517c950b0d87e7979a6f2f9f0979e4e053ff180daa not found: ID does not exist"
Jul 28 01:55:49.536799 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.536776    2578 scope.go:117] "RemoveContainer" containerID="2dc04364a5f28f670f270f3c8777628504bc3473e743acd4c396b05ca4913c54"
Jul 28 01:55:49.536991 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:55:49.536975    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc04364a5f28f670f270f3c8777628504bc3473e743acd4c396b05ca4913c54\": container with ID starting with 2dc04364a5f28f670f270f3c8777628504bc3473e743acd4c396b05ca4913c54 not found: ID does not exist" containerID="2dc04364a5f28f670f270f3c8777628504bc3473e743acd4c396b05ca4913c54"
Jul 28 01:55:49.537040 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.536996    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc04364a5f28f670f270f3c8777628504bc3473e743acd4c396b05ca4913c54"} err="failed to get container status \"2dc04364a5f28f670f270f3c8777628504bc3473e743acd4c396b05ca4913c54\": rpc error: code = NotFound desc = could not find container \"2dc04364a5f28f670f270f3c8777628504bc3473e743acd4c396b05ca4913c54\": container with ID starting with 2dc04364a5f28f670f270f3c8777628504bc3473e743acd4c396b05ca4913c54 not found: ID does not exist"
Jul 28 01:55:49.544465 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.544441    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"]
Jul 28 01:55:49.549840 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:49.549819    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-kzwmm"]
Jul 28 01:55:50.984833 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:55:50.984801    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84a250a9-58f2-4a12-8105-f546b8a85a02" path="/var/lib/kubelet/pods/84a250a9-58f2-4a12-8105-f546b8a85a02/volumes"
Jul 28 01:58:11.739471 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.739433    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"]
Jul 28 01:58:11.740008 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.739738    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84a250a9-58f2-4a12-8105-f546b8a85a02" containerName="storage-initializer"
Jul 28 01:58:11.740008 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.739749    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="84a250a9-58f2-4a12-8105-f546b8a85a02" containerName="storage-initializer"
Jul 28 01:58:11.740008 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.739760    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84a250a9-58f2-4a12-8105-f546b8a85a02" containerName="main"
Jul 28 01:58:11.740008 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.739767    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="84a250a9-58f2-4a12-8105-f546b8a85a02" containerName="main"
Jul 28 01:58:11.740008 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.739822    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="84a250a9-58f2-4a12-8105-f546b8a85a02" containerName="main"
Jul 28 01:58:11.741626 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.741610    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:11.745680 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.745659    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 28 01:58:11.745821 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.745660    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 28 01:58:11.745821 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.745660    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-6js2f\""
Jul 28 01:58:11.745821 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.745660    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 28 01:58:11.754372 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.754325    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"]
Jul 28 01:58:11.842539 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.842502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:11.842728 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.842561    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e58c52a-c23c-4717-ae09-9db40e95d611-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:11.842728 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.842609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:11.842728 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.842675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:11.842851 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.842729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:11.842851 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.842765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5vjv\" (UniqueName: \"kubernetes.io/projected/7e58c52a-c23c-4717-ae09-9db40e95d611-kube-api-access-h5vjv\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:11.943852 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.943815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5vjv\" (UniqueName: \"kubernetes.io/projected/7e58c52a-c23c-4717-ae09-9db40e95d611-kube-api-access-h5vjv\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:11.943852 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.943852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:11.944124 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.943879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e58c52a-c23c-4717-ae09-9db40e95d611-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:11.944124 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.943905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:11.944124 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.943943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:11.944124 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.943969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:11.944394 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.944363    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:11.944394 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.944387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:11.944494 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.944416    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:11.946421 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.946400    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:11.946581 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.946555    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e58c52a-c23c-4717-ae09-9db40e95d611-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:11.952358 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:11.952322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5vjv\" (UniqueName: \"kubernetes.io/projected/7e58c52a-c23c-4717-ae09-9db40e95d611-kube-api-access-h5vjv\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:12.052864 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:12.052823    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:12.189641 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:12.189613    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"]
Jul 28 01:58:12.192234 ip-10-0-139-81 kubenswrapper[2578]: W0728 01:58:12.192207    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e58c52a_c23c_4717_ae09_9db40e95d611.slice/crio-10b2aa163c1d76d3c1286c6d086c619c3670b9cd63dbb2d1e0083c3b6104911e WatchSource:0}: Error finding container 10b2aa163c1d76d3c1286c6d086c619c3670b9cd63dbb2d1e0083c3b6104911e: Status 404 returned error can't find the container with id 10b2aa163c1d76d3c1286c6d086c619c3670b9cd63dbb2d1e0083c3b6104911e
Jul 28 01:58:13.125361 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:13.125316    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9" event={"ID":"7e58c52a-c23c-4717-ae09-9db40e95d611","Type":"ContainerStarted","Data":"f204dc50edfbd96c9ae9968c72fbb01ddbcef2d3bdab1ee6d50e1bfd1a2a2366"}
Jul 28 01:58:13.125834 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:13.125390    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9" event={"ID":"7e58c52a-c23c-4717-ae09-9db40e95d611","Type":"ContainerStarted","Data":"10b2aa163c1d76d3c1286c6d086c619c3670b9cd63dbb2d1e0083c3b6104911e"}
Jul 28 01:58:13.149585 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:13.149532    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9" podStartSLOduration=2.149516214 podStartE2EDuration="2.149516214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:58:13.147363653 +0000 UTC m=+1936.724821269" watchObservedRunningTime="2026-07-28 01:58:13.149516214 +0000 UTC m=+1936.726973809"
Jul 28 01:58:14.053692 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:14.053660    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:22.053550 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:22.053510    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:22.066191 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:22.066162    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:22.181207 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:22.181180    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:45.265965 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.265931    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"]
Jul 28 01:58:45.275892 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:58:45.275858    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 01:58:45.275892 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:58:45.275892 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:58:45.275892 ip-10-0-139-81 kubenswrapper[2578]:  > containerID="f204dc50edfbd96c9ae9968c72fbb01ddbcef2d3bdab1ee6d50e1bfd1a2a2366" cmd=["/bin/sleep","15"]
Jul 28 01:58:45.276174 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.275906    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9" message="" err=<
Jul 28 01:58:45.276174 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:58:45.276174 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:58:45.276174 ip-10-0-139-81 kubenswrapper[2578]:  >
Jul 28 01:58:45.276174 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:58:45.275918    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 01:58:45.276174 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 01:58:45.276174 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 01:58:45.276174 ip-10-0-139-81 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9" podUID="7e58c52a-c23c-4717-ae09-9db40e95d611" containerName="main" containerID="cri-o://f204dc50edfbd96c9ae9968c72fbb01ddbcef2d3bdab1ee6d50e1bfd1a2a2366"
Jul 28 01:58:45.276174 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.275963    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9" podUID="7e58c52a-c23c-4717-ae09-9db40e95d611" containerName="main" containerID="cri-o://f204dc50edfbd96c9ae9968c72fbb01ddbcef2d3bdab1ee6d50e1bfd1a2a2366" gracePeriod=60
Jul 28 01:58:45.634231 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.634197    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:45.743454 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.743418    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-dshm\") pod \"7e58c52a-c23c-4717-ae09-9db40e95d611\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") "
Jul 28 01:58:45.743454 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.743453    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-tmp-dir\") pod \"7e58c52a-c23c-4717-ae09-9db40e95d611\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") "
Jul 28 01:58:45.743697 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.743497    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5vjv\" (UniqueName: \"kubernetes.io/projected/7e58c52a-c23c-4717-ae09-9db40e95d611-kube-api-access-h5vjv\") pod \"7e58c52a-c23c-4717-ae09-9db40e95d611\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") "
Jul 28 01:58:45.743697 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.743540    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-home\") pod \"7e58c52a-c23c-4717-ae09-9db40e95d611\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") "
Jul 28 01:58:45.743697 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.743582    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e58c52a-c23c-4717-ae09-9db40e95d611-tls-certs\") pod \"7e58c52a-c23c-4717-ae09-9db40e95d611\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") "
Jul 28 01:58:45.743697 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.743613    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-model-cache\") pod \"7e58c52a-c23c-4717-ae09-9db40e95d611\" (UID: \"7e58c52a-c23c-4717-ae09-9db40e95d611\") "
Jul 28 01:58:45.743929 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.743746    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-home" (OuterVolumeSpecName: "home") pod "7e58c52a-c23c-4717-ae09-9db40e95d611" (UID: "7e58c52a-c23c-4717-ae09-9db40e95d611"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:58:45.743929 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.743790    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e58c52a-c23c-4717-ae09-9db40e95d611" (UID: "7e58c52a-c23c-4717-ae09-9db40e95d611"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:58:45.744028 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.743943    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e58c52a-c23c-4717-ae09-9db40e95d611" (UID: "7e58c52a-c23c-4717-ae09-9db40e95d611"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:58:45.744028 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.743945    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:58:45.744028 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.743991    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:58:45.745790 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.745763    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-dshm" (OuterVolumeSpecName: "dshm") pod "7e58c52a-c23c-4717-ae09-9db40e95d611" (UID: "7e58c52a-c23c-4717-ae09-9db40e95d611"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:58:45.745902 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.745847    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e58c52a-c23c-4717-ae09-9db40e95d611-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e58c52a-c23c-4717-ae09-9db40e95d611" (UID: "7e58c52a-c23c-4717-ae09-9db40e95d611"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:58:45.745902 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.745871    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e58c52a-c23c-4717-ae09-9db40e95d611-kube-api-access-h5vjv" (OuterVolumeSpecName: "kube-api-access-h5vjv") pod "7e58c52a-c23c-4717-ae09-9db40e95d611" (UID: "7e58c52a-c23c-4717-ae09-9db40e95d611"). InnerVolumeSpecName "kube-api-access-h5vjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:58:45.845101 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.845001    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:58:45.845101 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.845034    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5vjv\" (UniqueName: \"kubernetes.io/projected/7e58c52a-c23c-4717-ae09-9db40e95d611-kube-api-access-h5vjv\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:58:45.845101 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.845045    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e58c52a-c23c-4717-ae09-9db40e95d611-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:58:45.845101 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:45.845054    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e58c52a-c23c-4717-ae09-9db40e95d611-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 01:58:46.279194 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:46.279159    2578 generic.go:356] "Generic (PLEG): container finished" podID="7e58c52a-c23c-4717-ae09-9db40e95d611" containerID="f204dc50edfbd96c9ae9968c72fbb01ddbcef2d3bdab1ee6d50e1bfd1a2a2366" exitCode=0
Jul 28 01:58:46.279619 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:46.279211    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"
Jul 28 01:58:46.279619 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:46.279234    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9" event={"ID":"7e58c52a-c23c-4717-ae09-9db40e95d611","Type":"ContainerDied","Data":"f204dc50edfbd96c9ae9968c72fbb01ddbcef2d3bdab1ee6d50e1bfd1a2a2366"}
Jul 28 01:58:46.279619 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:46.279285    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9" event={"ID":"7e58c52a-c23c-4717-ae09-9db40e95d611","Type":"ContainerDied","Data":"10b2aa163c1d76d3c1286c6d086c619c3670b9cd63dbb2d1e0083c3b6104911e"}
Jul 28 01:58:46.279619 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:46.279317    2578 scope.go:117] "RemoveContainer" containerID="f204dc50edfbd96c9ae9968c72fbb01ddbcef2d3bdab1ee6d50e1bfd1a2a2366"
Jul 28 01:58:46.292897 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:46.292879    2578 scope.go:117] "RemoveContainer" containerID="f204dc50edfbd96c9ae9968c72fbb01ddbcef2d3bdab1ee6d50e1bfd1a2a2366"
Jul 28 01:58:46.293163 ip-10-0-139-81 kubenswrapper[2578]: E0728 01:58:46.293144    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f204dc50edfbd96c9ae9968c72fbb01ddbcef2d3bdab1ee6d50e1bfd1a2a2366\": container with ID starting with f204dc50edfbd96c9ae9968c72fbb01ddbcef2d3bdab1ee6d50e1bfd1a2a2366 not found: ID does not exist" containerID="f204dc50edfbd96c9ae9968c72fbb01ddbcef2d3bdab1ee6d50e1bfd1a2a2366"
Jul 28 01:58:46.293227 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:46.293173    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f204dc50edfbd96c9ae9968c72fbb01ddbcef2d3bdab1ee6d50e1bfd1a2a2366"} err="failed to get container status \"f204dc50edfbd96c9ae9968c72fbb01ddbcef2d3bdab1ee6d50e1bfd1a2a2366\": rpc error: code = NotFound desc = could not find container \"f204dc50edfbd96c9ae9968c72fbb01ddbcef2d3bdab1ee6d50e1bfd1a2a2366\": container with ID starting with f204dc50edfbd96c9ae9968c72fbb01ddbcef2d3bdab1ee6d50e1bfd1a2a2366 not found: ID does not exist"
Jul 28 01:58:46.306168 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:46.306144    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"]
Jul 28 01:58:46.309962 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:46.309941    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954d9k9"]
Jul 28 01:58:46.985461 ip-10-0-139-81 kubenswrapper[2578]: I0728 01:58:46.985430    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e58c52a-c23c-4717-ae09-9db40e95d611" path="/var/lib/kubelet/pods/7e58c52a-c23c-4717-ae09-9db40e95d611/volumes"
Jul 28 02:00:19.233496 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.233418    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"]
Jul 28 02:00:19.233938 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.233722    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e58c52a-c23c-4717-ae09-9db40e95d611" containerName="main"
Jul 28 02:00:19.233938 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.233732    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e58c52a-c23c-4717-ae09-9db40e95d611" containerName="main"
Jul 28 02:00:19.233938 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.233781    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e58c52a-c23c-4717-ae09-9db40e95d611" containerName="main"
Jul 28 02:00:19.236543 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.236526    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.240900 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.240876    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 28 02:00:19.241043 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.240922    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 28 02:00:19.241043 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.240943    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-pgwkj\""
Jul 28 02:00:19.241043 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.240995    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 28 02:00:19.247256 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.247234    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"]
Jul 28 02:00:19.296314 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.296270    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.296314 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.296316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.296553 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.296373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hzlt\" (UniqueName: \"kubernetes.io/projected/26ec09fd-da94-464d-b65a-6fce34f51fe0-kube-api-access-9hzlt\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.296553 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.296422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.296553 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.296444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.296553 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.296500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26ec09fd-da94-464d-b65a-6fce34f51fe0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.397680 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.397638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.397680 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.397686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hzlt\" (UniqueName: \"kubernetes.io/projected/26ec09fd-da94-464d-b65a-6fce34f51fe0-kube-api-access-9hzlt\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.397931 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.397744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.397931 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.397771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.397931 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.397810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26ec09fd-da94-464d-b65a-6fce34f51fe0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.397931 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.397873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.398168 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.398147    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.398231 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.398180    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.398231 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.398219    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.400099 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.400079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.400361 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.400326    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26ec09fd-da94-464d-b65a-6fce34f51fe0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.407087 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.407065    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hzlt\" (UniqueName: \"kubernetes.io/projected/26ec09fd-da94-464d-b65a-6fce34f51fe0-kube-api-access-9hzlt\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.547016 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.546973    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:19.680731 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:19.680702    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"]
Jul 28 02:00:19.683453 ip-10-0-139-81 kubenswrapper[2578]: W0728 02:00:19.683424    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ec09fd_da94_464d_b65a_6fce34f51fe0.slice/crio-56493a109f9efcb0e4d4c19f6da30a46c26277ba9cc36a9af073e6d0080c665c WatchSource:0}: Error finding container 56493a109f9efcb0e4d4c19f6da30a46c26277ba9cc36a9af073e6d0080c665c: Status 404 returned error can't find the container with id 56493a109f9efcb0e4d4c19f6da30a46c26277ba9cc36a9af073e6d0080c665c
Jul 28 02:00:20.672467 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:20.672435    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8" event={"ID":"26ec09fd-da94-464d-b65a-6fce34f51fe0","Type":"ContainerStarted","Data":"3b53c4ce51362d64ed341e1880e462520e62ef698933c0de3957266778ba4067"}
Jul 28 02:00:20.672861 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:20.672486    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8" event={"ID":"26ec09fd-da94-464d-b65a-6fce34f51fe0","Type":"ContainerStarted","Data":"56493a109f9efcb0e4d4c19f6da30a46c26277ba9cc36a9af073e6d0080c665c"}
Jul 28 02:00:20.693797 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:20.693752    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8" podStartSLOduration=1.693736558 podStartE2EDuration="1.693736558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:00:20.69138016 +0000 UTC m=+2064.268837776" watchObservedRunningTime="2026-07-28 02:00:20.693736558 +0000 UTC m=+2064.271194174"
Jul 28 02:00:21.547133 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:21.547102    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:29.547491 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:29.547454    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:29.559899 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:29.559873    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:29.723046 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:29.723022    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:58.172625 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.172592    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"]
Jul 28 02:00:58.181391 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:00:58.181352    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 02:00:58.181391 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:00:58.181391 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 02:00:58.181391 ip-10-0-139-81 kubenswrapper[2578]:  > containerID="3b53c4ce51362d64ed341e1880e462520e62ef698933c0de3957266778ba4067" cmd=["/bin/sleep","15"]
Jul 28 02:00:58.181682 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.181414    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8" message="" err=<
Jul 28 02:00:58.181682 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:00:58.181682 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 02:00:58.181682 ip-10-0-139-81 kubenswrapper[2578]:  >
Jul 28 02:00:58.181682 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:00:58.181430    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 02:00:58.181682 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:00:58.181682 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 02:00:58.181682 ip-10-0-139-81 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8" podUID="26ec09fd-da94-464d-b65a-6fce34f51fe0" containerName="main" containerID="cri-o://3b53c4ce51362d64ed341e1880e462520e62ef698933c0de3957266778ba4067"
Jul 28 02:00:58.181682 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.181488    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8" podUID="26ec09fd-da94-464d-b65a-6fce34f51fe0" containerName="main" containerID="cri-o://3b53c4ce51362d64ed341e1880e462520e62ef698933c0de3957266778ba4067" gracePeriod=60
Jul 28 02:00:58.529820 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.529798    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:58.728643 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.728549    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26ec09fd-da94-464d-b65a-6fce34f51fe0-tls-certs\") pod \"26ec09fd-da94-464d-b65a-6fce34f51fe0\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") "
Jul 28 02:00:58.728643 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.728622    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hzlt\" (UniqueName: \"kubernetes.io/projected/26ec09fd-da94-464d-b65a-6fce34f51fe0-kube-api-access-9hzlt\") pod \"26ec09fd-da94-464d-b65a-6fce34f51fe0\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") "
Jul 28 02:00:58.728860 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.728654    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-dshm\") pod \"26ec09fd-da94-464d-b65a-6fce34f51fe0\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") "
Jul 28 02:00:58.728860 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.728680    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-home\") pod \"26ec09fd-da94-464d-b65a-6fce34f51fe0\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") "
Jul 28 02:00:58.728860 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.728703    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-model-cache\") pod \"26ec09fd-da94-464d-b65a-6fce34f51fe0\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") "
Jul 28 02:00:58.728860 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.728763    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-tmp-dir\") pod \"26ec09fd-da94-464d-b65a-6fce34f51fe0\" (UID: \"26ec09fd-da94-464d-b65a-6fce34f51fe0\") "
Jul 28 02:00:58.729086 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.728976    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-home" (OuterVolumeSpecName: "home") pod "26ec09fd-da94-464d-b65a-6fce34f51fe0" (UID: "26ec09fd-da94-464d-b65a-6fce34f51fe0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:00:58.729172 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.729141    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "26ec09fd-da94-464d-b65a-6fce34f51fe0" (UID: "26ec09fd-da94-464d-b65a-6fce34f51fe0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:00:58.729317 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.729194    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-model-cache" (OuterVolumeSpecName: "model-cache") pod "26ec09fd-da94-464d-b65a-6fce34f51fe0" (UID: "26ec09fd-da94-464d-b65a-6fce34f51fe0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:00:58.730901 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.730876    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ec09fd-da94-464d-b65a-6fce34f51fe0-kube-api-access-9hzlt" (OuterVolumeSpecName: "kube-api-access-9hzlt") pod "26ec09fd-da94-464d-b65a-6fce34f51fe0" (UID: "26ec09fd-da94-464d-b65a-6fce34f51fe0"). InnerVolumeSpecName "kube-api-access-9hzlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:00:58.731008 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.730916    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ec09fd-da94-464d-b65a-6fce34f51fe0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26ec09fd-da94-464d-b65a-6fce34f51fe0" (UID: "26ec09fd-da94-464d-b65a-6fce34f51fe0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:00:58.731008 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.730958    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-dshm" (OuterVolumeSpecName: "dshm") pod "26ec09fd-da94-464d-b65a-6fce34f51fe0" (UID: "26ec09fd-da94-464d-b65a-6fce34f51fe0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:00:58.829716 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.829682    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26ec09fd-da94-464d-b65a-6fce34f51fe0-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:00:58.829716 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.829708    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hzlt\" (UniqueName: \"kubernetes.io/projected/26ec09fd-da94-464d-b65a-6fce34f51fe0-kube-api-access-9hzlt\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:00:58.829716 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.829721    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:00:58.829999 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.829753    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:00:58.829999 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.829766    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:00:58.829999 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.829780    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26ec09fd-da94-464d-b65a-6fce34f51fe0-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:00:58.834566 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.834537    2578 generic.go:356] "Generic (PLEG): container finished" podID="26ec09fd-da94-464d-b65a-6fce34f51fe0" containerID="3b53c4ce51362d64ed341e1880e462520e62ef698933c0de3957266778ba4067" exitCode=0
Jul 28 02:00:58.834707 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.834580    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"
Jul 28 02:00:58.834707 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.834583    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8" event={"ID":"26ec09fd-da94-464d-b65a-6fce34f51fe0","Type":"ContainerDied","Data":"3b53c4ce51362d64ed341e1880e462520e62ef698933c0de3957266778ba4067"}
Jul 28 02:00:58.834822 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.834725    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8" event={"ID":"26ec09fd-da94-464d-b65a-6fce34f51fe0","Type":"ContainerDied","Data":"56493a109f9efcb0e4d4c19f6da30a46c26277ba9cc36a9af073e6d0080c665c"}
Jul 28 02:00:58.834822 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.834768    2578 scope.go:117] "RemoveContainer" containerID="3b53c4ce51362d64ed341e1880e462520e62ef698933c0de3957266778ba4067"
Jul 28 02:00:58.845944 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.845926    2578 scope.go:117] "RemoveContainer" containerID="3b53c4ce51362d64ed341e1880e462520e62ef698933c0de3957266778ba4067"
Jul 28 02:00:58.846212 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:00:58.846193    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b53c4ce51362d64ed341e1880e462520e62ef698933c0de3957266778ba4067\": container with ID starting with 3b53c4ce51362d64ed341e1880e462520e62ef698933c0de3957266778ba4067 not found: ID does not exist" containerID="3b53c4ce51362d64ed341e1880e462520e62ef698933c0de3957266778ba4067"
Jul 28 02:00:58.846277 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.846224    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b53c4ce51362d64ed341e1880e462520e62ef698933c0de3957266778ba4067"} err="failed to get container status \"3b53c4ce51362d64ed341e1880e462520e62ef698933c0de3957266778ba4067\": rpc error: code = NotFound desc = could not find container \"3b53c4ce51362d64ed341e1880e462520e62ef698933c0de3957266778ba4067\": container with ID starting with 3b53c4ce51362d64ed341e1880e462520e62ef698933c0de3957266778ba4067 not found: ID does not exist"
Jul 28 02:00:58.861263 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.861241    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"]
Jul 28 02:00:58.865107 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.865087    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-cshq8"]
Jul 28 02:00:58.987838 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:00:58.987741    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26ec09fd-da94-464d-b65a-6fce34f51fe0" path="/var/lib/kubelet/pods/26ec09fd-da94-464d-b65a-6fce34f51fe0/volumes"
Jul 28 02:05:05.519217 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:05.519133    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc"]
Jul 28 02:05:10.122880 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:10.122832    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc" podUID="4724367b-3420-49c1-8388-f26595737bd5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 02:05:20.122806 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:20.122760    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc" podUID="4724367b-3420-49c1-8388-f26595737bd5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 02:05:20.538481 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:20.538438    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc" podUID="4724367b-3420-49c1-8388-f26595737bd5" containerName="main" containerID="cri-o://cfb2c7ba6462e17d7b133de5ebad6eb43d7743769edde2241e523cc5284cdf31" gracePeriod=45
Jul 28 02:05:20.791769 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:20.791706    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc"
Jul 28 02:05:20.890663 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:20.890629    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgk4v\" (UniqueName: \"kubernetes.io/projected/4724367b-3420-49c1-8388-f26595737bd5-kube-api-access-qgk4v\") pod \"4724367b-3420-49c1-8388-f26595737bd5\" (UID: \"4724367b-3420-49c1-8388-f26595737bd5\") "
Jul 28 02:05:20.890844 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:20.890680    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4724367b-3420-49c1-8388-f26595737bd5-tls-certs\") pod \"4724367b-3420-49c1-8388-f26595737bd5\" (UID: \"4724367b-3420-49c1-8388-f26595737bd5\") "
Jul 28 02:05:20.892931 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:20.892897    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4724367b-3420-49c1-8388-f26595737bd5-kube-api-access-qgk4v" (OuterVolumeSpecName: "kube-api-access-qgk4v") pod "4724367b-3420-49c1-8388-f26595737bd5" (UID: "4724367b-3420-49c1-8388-f26595737bd5"). InnerVolumeSpecName "kube-api-access-qgk4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:05:20.892931 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:20.892899    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4724367b-3420-49c1-8388-f26595737bd5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4724367b-3420-49c1-8388-f26595737bd5" (UID: "4724367b-3420-49c1-8388-f26595737bd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:05:20.975103 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:20.975067    2578 generic.go:356] "Generic (PLEG): container finished" podID="4724367b-3420-49c1-8388-f26595737bd5" containerID="cfb2c7ba6462e17d7b133de5ebad6eb43d7743769edde2241e523cc5284cdf31" exitCode=0
Jul 28 02:05:20.975309 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:20.975123    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc" event={"ID":"4724367b-3420-49c1-8388-f26595737bd5","Type":"ContainerDied","Data":"cfb2c7ba6462e17d7b133de5ebad6eb43d7743769edde2241e523cc5284cdf31"}
Jul 28 02:05:20.975309 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:20.975133    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc"
Jul 28 02:05:20.975309 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:20.975187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc" event={"ID":"4724367b-3420-49c1-8388-f26595737bd5","Type":"ContainerDied","Data":"aa75fb276ac0f08e09d337c71a9dfc52cd2c7d6b93b5004dde3d6d8a3c16fd24"}
Jul 28 02:05:20.975309 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:20.975209    2578 scope.go:117] "RemoveContainer" containerID="cfb2c7ba6462e17d7b133de5ebad6eb43d7743769edde2241e523cc5284cdf31"
Jul 28 02:05:20.983662 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:20.983633    2578 scope.go:117] "RemoveContainer" containerID="cfb2c7ba6462e17d7b133de5ebad6eb43d7743769edde2241e523cc5284cdf31"
Jul 28 02:05:20.983975 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:05:20.983945    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfb2c7ba6462e17d7b133de5ebad6eb43d7743769edde2241e523cc5284cdf31\": container with ID starting with cfb2c7ba6462e17d7b133de5ebad6eb43d7743769edde2241e523cc5284cdf31 not found: ID does not exist" containerID="cfb2c7ba6462e17d7b133de5ebad6eb43d7743769edde2241e523cc5284cdf31"
Jul 28 02:05:20.984092 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:20.983984    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb2c7ba6462e17d7b133de5ebad6eb43d7743769edde2241e523cc5284cdf31"} err="failed to get container status \"cfb2c7ba6462e17d7b133de5ebad6eb43d7743769edde2241e523cc5284cdf31\": rpc error: code = NotFound desc = could not find container \"cfb2c7ba6462e17d7b133de5ebad6eb43d7743769edde2241e523cc5284cdf31\": container with ID starting with cfb2c7ba6462e17d7b133de5ebad6eb43d7743769edde2241e523cc5284cdf31 not found: ID does not exist"
Jul 28 02:05:20.991343 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:20.991309    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgk4v\" (UniqueName: \"kubernetes.io/projected/4724367b-3420-49c1-8388-f26595737bd5-kube-api-access-qgk4v\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:05:20.991434 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:20.991348    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4724367b-3420-49c1-8388-f26595737bd5-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:05:20.997625 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:20.997599    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc"]
Jul 28 02:05:20.999954 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:20.999934    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-router-scheduler-5bbdf87d58-gblfc"]
Jul 28 02:05:22.986135 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:22.986102    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4724367b-3420-49c1-8388-f26595737bd5" path="/var/lib/kubelet/pods/4724367b-3420-49c1-8388-f26595737bd5/volumes"
Jul 28 02:05:36.929788 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:36.929753    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"]
Jul 28 02:05:36.930429 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:36.930079    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4724367b-3420-49c1-8388-f26595737bd5" containerName="main"
Jul 28 02:05:36.930429 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:36.930091    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4724367b-3420-49c1-8388-f26595737bd5" containerName="main"
Jul 28 02:05:36.930429 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:36.930107    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ec09fd-da94-464d-b65a-6fce34f51fe0" containerName="main"
Jul 28 02:05:36.930429 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:36.930113    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ec09fd-da94-464d-b65a-6fce34f51fe0" containerName="main"
Jul 28 02:05:36.930429 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:36.930163    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4724367b-3420-49c1-8388-f26595737bd5" containerName="main"
Jul 28 02:05:36.930429 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:36.930173    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ec09fd-da94-464d-b65a-6fce34f51fe0" containerName="main"
Jul 28 02:05:36.932969 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:36.932948    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:36.935856 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:36.935835    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"kube-root-ca.crt\""
Jul 28 02:05:36.937226 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:36.937210    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"openshift-service-ca.crt\""
Jul 28 02:05:36.937304 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:36.937287    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"default-dockercfg-86qlh\""
Jul 28 02:05:36.937380 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:36.937321    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"autoscale-keda-deploy-kserve-self-signed-certs\""
Jul 28 02:05:36.943774 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:36.943749    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"]
Jul 28 02:05:37.020593 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.020562    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:37.020593 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.020597    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52c8w\" (UniqueName: \"kubernetes.io/projected/5b04c603-3180-4027-8f92-9c238db1c504-kube-api-access-52c8w\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:37.020814 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.020645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:37.020814 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.020675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:37.020814 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.020715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b04c603-3180-4027-8f92-9c238db1c504-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:37.020814 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.020757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:37.121606 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.121570    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b04c603-3180-4027-8f92-9c238db1c504-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:37.121818 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.121616    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:37.121818 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.121671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:37.121818 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.121697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52c8w\" (UniqueName: \"kubernetes.io/projected/5b04c603-3180-4027-8f92-9c238db1c504-kube-api-access-52c8w\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:37.121818 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.121745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:37.121818 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.121775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:37.122176 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.122156    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:37.122260 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.122155    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:37.122260 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.122209    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:37.123945 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.123927    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:37.124236 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.124215    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b04c603-3180-4027-8f92-9c238db1c504-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:37.129890 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.129865    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52c8w\" (UniqueName: \"kubernetes.io/projected/5b04c603-3180-4027-8f92-9c238db1c504-kube-api-access-52c8w\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:37.243762 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.243677    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:37.375638 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:37.375537    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"]
Jul 28 02:05:37.378002 ip-10-0-139-81 kubenswrapper[2578]: W0728 02:05:37.377928    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b04c603_3180_4027_8f92_9c238db1c504.slice/crio-21645829caa31b093a60867c73bcb8263f26e5e2d7f9cae607268707703926ca WatchSource:0}: Error finding container 21645829caa31b093a60867c73bcb8263f26e5e2d7f9cae607268707703926ca: Status 404 returned error can't find the container with id 21645829caa31b093a60867c73bcb8263f26e5e2d7f9cae607268707703926ca
Jul 28 02:05:38.048245 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:38.048209    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl" event={"ID":"5b04c603-3180-4027-8f92-9c238db1c504","Type":"ContainerStarted","Data":"85d0204edf60df300839fc7bcfea5c1c137ca6529eab2bacccf3c3a8d8bfe137"}
Jul 28 02:05:38.048774 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:38.048263    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl" event={"ID":"5b04c603-3180-4027-8f92-9c238db1c504","Type":"ContainerStarted","Data":"21645829caa31b093a60867c73bcb8263f26e5e2d7f9cae607268707703926ca"}
Jul 28 02:05:38.067478 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:38.067419    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl" podStartSLOduration=2.067397307 podStartE2EDuration="2.067397307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:05:38.065041535 +0000 UTC m=+2381.642499191" watchObservedRunningTime="2026-07-28 02:05:38.067397307 +0000 UTC m=+2381.644854925"
Jul 28 02:05:38.244174 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:38.244139    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:47.244105 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:47.244063    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:47.257152 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:47.257122    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:05:48.111948 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:05:48.111914    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:09:55.975916 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:55.975885    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w"]
Jul 28 02:09:55.979049 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:55.979031    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w"
Jul 28 02:09:55.982083 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:55.982060    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 28 02:09:55.983316 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:55.983284    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 28 02:09:55.983316 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:55.983301    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-fj8rq\""
Jul 28 02:09:55.983529 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:55.983510    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-ws6bz\""
Jul 28 02:09:55.983628 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:55.983609    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 28 02:09:55.988508 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:55.988487    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w"]
Jul 28 02:09:56.049307 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:56.049282    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/553127d1-2dd7-4c00-a50f-abdd6b190186-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w\" (UID: \"553127d1-2dd7-4c00-a50f-abdd6b190186\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w"
Jul 28 02:09:56.049436 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:56.049323    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmqqf\" (UniqueName: \"kubernetes.io/projected/553127d1-2dd7-4c00-a50f-abdd6b190186-kube-api-access-gmqqf\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w\" (UID: \"553127d1-2dd7-4c00-a50f-abdd6b190186\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w"
Jul 28 02:09:56.150125 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:56.150092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/553127d1-2dd7-4c00-a50f-abdd6b190186-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w\" (UID: \"553127d1-2dd7-4c00-a50f-abdd6b190186\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w"
Jul 28 02:09:56.150313 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:56.150143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmqqf\" (UniqueName: \"kubernetes.io/projected/553127d1-2dd7-4c00-a50f-abdd6b190186-kube-api-access-gmqqf\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w\" (UID: \"553127d1-2dd7-4c00-a50f-abdd6b190186\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w"
Jul 28 02:09:56.152603 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:56.152583    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/553127d1-2dd7-4c00-a50f-abdd6b190186-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w\" (UID: \"553127d1-2dd7-4c00-a50f-abdd6b190186\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w"
Jul 28 02:09:56.160517 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:56.160493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmqqf\" (UniqueName: \"kubernetes.io/projected/553127d1-2dd7-4c00-a50f-abdd6b190186-kube-api-access-gmqqf\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w\" (UID: \"553127d1-2dd7-4c00-a50f-abdd6b190186\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w"
Jul 28 02:09:56.289937 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:56.289906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w"
Jul 28 02:09:56.420149 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:56.420123    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w"]
Jul 28 02:09:56.422261 ip-10-0-139-81 kubenswrapper[2578]: W0728 02:09:56.422232    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553127d1_2dd7_4c00_a50f_abdd6b190186.slice/crio-094203577b84ae3461fe7af680549ac84114693752a42cf3aec633061d644ddb WatchSource:0}: Error finding container 094203577b84ae3461fe7af680549ac84114693752a42cf3aec633061d644ddb: Status 404 returned error can't find the container with id 094203577b84ae3461fe7af680549ac84114693752a42cf3aec633061d644ddb
Jul 28 02:09:57.148157 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:57.148121    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w" event={"ID":"553127d1-2dd7-4c00-a50f-abdd6b190186","Type":"ContainerStarted","Data":"251339ca2820ecbb0b66fcc0fb8d3ab1678779ca464b918dd7123e7379228852"}
Jul 28 02:09:57.148643 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:57.148179    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w" event={"ID":"553127d1-2dd7-4c00-a50f-abdd6b190186","Type":"ContainerStarted","Data":"094203577b84ae3461fe7af680549ac84114693752a42cf3aec633061d644ddb"}
Jul 28 02:09:57.148643 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:57.148205    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w"
Jul 28 02:09:57.168854 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:09:57.168804    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w" podStartSLOduration=2.168786723 podStartE2EDuration="2.168786723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:09:57.168056614 +0000 UTC m=+2640.745514230" watchObservedRunningTime="2026-07-28 02:09:57.168786723 +0000 UTC m=+2640.746244339"
Jul 28 02:10:28.155107 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:10:28.155077    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w"
Jul 28 02:20:44.307128 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.307094    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"]
Jul 28 02:20:44.316058 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:20:44.316013    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 02:20:44.316058 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:20:44.316058 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 02:20:44.316058 ip-10-0-139-81 kubenswrapper[2578]:  > containerID="85d0204edf60df300839fc7bcfea5c1c137ca6529eab2bacccf3c3a8d8bfe137" cmd=["/bin/sleep","15"]
Jul 28 02:20:44.316357 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.316085    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl" message="" err=<
Jul 28 02:20:44.316357 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:20:44.316357 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 02:20:44.316357 ip-10-0-139-81 kubenswrapper[2578]:  >
Jul 28 02:20:44.316357 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:20:44.316104    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 02:20:44.316357 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:20:44.316357 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 02:20:44.316357 ip-10-0-139-81 kubenswrapper[2578]:  > pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl" podUID="5b04c603-3180-4027-8f92-9c238db1c504" containerName="main" containerID="cri-o://85d0204edf60df300839fc7bcfea5c1c137ca6529eab2bacccf3c3a8d8bfe137"
Jul 28 02:20:44.316357 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.316164    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl" podUID="5b04c603-3180-4027-8f92-9c238db1c504" containerName="main" containerID="cri-o://85d0204edf60df300839fc7bcfea5c1c137ca6529eab2bacccf3c3a8d8bfe137" gracePeriod=60
Jul 28 02:20:44.661469 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.661443    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:20:44.692789 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.692766    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-dshm\") pod \"5b04c603-3180-4027-8f92-9c238db1c504\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") "
Jul 28 02:20:44.692969 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.692833    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-tmp-dir\") pod \"5b04c603-3180-4027-8f92-9c238db1c504\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") "
Jul 28 02:20:44.692969 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.692851    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-model-cache\") pod \"5b04c603-3180-4027-8f92-9c238db1c504\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") "
Jul 28 02:20:44.692969 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.692867    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b04c603-3180-4027-8f92-9c238db1c504-tls-certs\") pod \"5b04c603-3180-4027-8f92-9c238db1c504\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") "
Jul 28 02:20:44.692969 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.692900    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52c8w\" (UniqueName: \"kubernetes.io/projected/5b04c603-3180-4027-8f92-9c238db1c504-kube-api-access-52c8w\") pod \"5b04c603-3180-4027-8f92-9c238db1c504\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") "
Jul 28 02:20:44.692969 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.692915    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-home\") pod \"5b04c603-3180-4027-8f92-9c238db1c504\" (UID: \"5b04c603-3180-4027-8f92-9c238db1c504\") "
Jul 28 02:20:44.693254 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.693101    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b04c603-3180-4027-8f92-9c238db1c504" (UID: "5b04c603-3180-4027-8f92-9c238db1c504"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:20:44.693254 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.693110    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b04c603-3180-4027-8f92-9c238db1c504" (UID: "5b04c603-3180-4027-8f92-9c238db1c504"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:20:44.693377 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.693304    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-home" (OuterVolumeSpecName: "home") pod "5b04c603-3180-4027-8f92-9c238db1c504" (UID: "5b04c603-3180-4027-8f92-9c238db1c504"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:20:44.695088 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.695061    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-dshm" (OuterVolumeSpecName: "dshm") pod "5b04c603-3180-4027-8f92-9c238db1c504" (UID: "5b04c603-3180-4027-8f92-9c238db1c504"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:20:44.695088 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.695080    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b04c603-3180-4027-8f92-9c238db1c504-kube-api-access-52c8w" (OuterVolumeSpecName: "kube-api-access-52c8w") pod "5b04c603-3180-4027-8f92-9c238db1c504" (UID: "5b04c603-3180-4027-8f92-9c238db1c504"). InnerVolumeSpecName "kube-api-access-52c8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:20:44.695239 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.695153    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b04c603-3180-4027-8f92-9c238db1c504-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b04c603-3180-4027-8f92-9c238db1c504" (UID: "5b04c603-3180-4027-8f92-9c238db1c504"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:20:44.793891 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.793861    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52c8w\" (UniqueName: \"kubernetes.io/projected/5b04c603-3180-4027-8f92-9c238db1c504-kube-api-access-52c8w\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:20:44.793891 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.793887    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:20:44.793891 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.793896    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:20:44.793891 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.793903    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:20:44.794165 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.793912    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b04c603-3180-4027-8f92-9c238db1c504-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:20:44.794165 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.793920    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b04c603-3180-4027-8f92-9c238db1c504-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:20:44.888426 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.888320    2578 generic.go:356] "Generic (PLEG): container finished" podID="5b04c603-3180-4027-8f92-9c238db1c504" containerID="85d0204edf60df300839fc7bcfea5c1c137ca6529eab2bacccf3c3a8d8bfe137" exitCode=0
Jul 28 02:20:44.888426 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.888378    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"
Jul 28 02:20:44.888617 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.888457    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl" event={"ID":"5b04c603-3180-4027-8f92-9c238db1c504","Type":"ContainerDied","Data":"85d0204edf60df300839fc7bcfea5c1c137ca6529eab2bacccf3c3a8d8bfe137"}
Jul 28 02:20:44.888617 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.888496    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl" event={"ID":"5b04c603-3180-4027-8f92-9c238db1c504","Type":"ContainerDied","Data":"21645829caa31b093a60867c73bcb8263f26e5e2d7f9cae607268707703926ca"}
Jul 28 02:20:44.888617 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.888522    2578 scope.go:117] "RemoveContainer" containerID="85d0204edf60df300839fc7bcfea5c1c137ca6529eab2bacccf3c3a8d8bfe137"
Jul 28 02:20:44.897689 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.897671    2578 scope.go:117] "RemoveContainer" containerID="85d0204edf60df300839fc7bcfea5c1c137ca6529eab2bacccf3c3a8d8bfe137"
Jul 28 02:20:44.897987 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:20:44.897961    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d0204edf60df300839fc7bcfea5c1c137ca6529eab2bacccf3c3a8d8bfe137\": container with ID starting with 85d0204edf60df300839fc7bcfea5c1c137ca6529eab2bacccf3c3a8d8bfe137 not found: ID does not exist" containerID="85d0204edf60df300839fc7bcfea5c1c137ca6529eab2bacccf3c3a8d8bfe137"
Jul 28 02:20:44.898086 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.897994    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d0204edf60df300839fc7bcfea5c1c137ca6529eab2bacccf3c3a8d8bfe137"} err="failed to get container status \"85d0204edf60df300839fc7bcfea5c1c137ca6529eab2bacccf3c3a8d8bfe137\": rpc error: code = NotFound desc = could not find container \"85d0204edf60df300839fc7bcfea5c1c137ca6529eab2bacccf3c3a8d8bfe137\": container with ID starting with 85d0204edf60df300839fc7bcfea5c1c137ca6529eab2bacccf3c3a8d8bfe137 not found: ID does not exist"
Jul 28 02:20:44.910147 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.910120    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"]
Jul 28 02:20:44.913488 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.913462    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-jkkpl"]
Jul 28 02:20:44.987162 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:20:44.987123    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b04c603-3180-4027-8f92-9c238db1c504" path="/var/lib/kubelet/pods/5b04c603-3180-4027-8f92-9c238db1c504/volumes"
Jul 28 02:21:26.177055 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.176971    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 28 02:21:26.177584 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.177266    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b04c603-3180-4027-8f92-9c238db1c504" containerName="main"
Jul 28 02:21:26.177584 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.177277    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b04c603-3180-4027-8f92-9c238db1c504" containerName="main"
Jul 28 02:21:26.177584 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.177360    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b04c603-3180-4027-8f92-9c238db1c504" containerName="main"
Jul 28 02:21:26.180301 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.180276    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.185681 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.185655    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"openshift-service-ca.crt\""
Jul 28 02:21:26.185843 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.185817    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-self-signed-certs\""
Jul 28 02:21:26.185916 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.185840    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-mn-dockercfg-5bgns\""
Jul 28 02:21:26.185972 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.185920    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"default-dockercfg-bg5dm\""
Jul 28 02:21:26.186028 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.185817    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"kube-root-ca.crt\""
Jul 28 02:21:26.193312 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.193286    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 28 02:21:26.331824 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.331788    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-home\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.331824 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.331821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.332072 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.331855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e296857-2009-48f2-9540-c8d94e468c88-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.332072 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.331920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7w8k\" (UniqueName: \"kubernetes.io/projected/7e296857-2009-48f2-9540-c8d94e468c88-kube-api-access-w7w8k\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.332072 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.331981    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.332072 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.332026    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.432822 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.432714    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-home\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.432822 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.432758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.432822 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.432787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e296857-2009-48f2-9540-c8d94e468c88-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.432822 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.432816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7w8k\" (UniqueName: \"kubernetes.io/projected/7e296857-2009-48f2-9540-c8d94e468c88-kube-api-access-w7w8k\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.433180 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.432868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.433180 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.432913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.433180 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.433137    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-home\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.433320 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.433236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.433320 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.433251    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.435080 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.435057    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.435185 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.435167    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e296857-2009-48f2-9540-c8d94e468c88-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.440563 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.440540    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7w8k\" (UniqueName: \"kubernetes.io/projected/7e296857-2009-48f2-9540-c8d94e468c88-kube-api-access-w7w8k\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.490854 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.490815    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:26.621030 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:26.621006    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 28 02:21:26.623227 ip-10-0-139-81 kubenswrapper[2578]: W0728 02:21:26.623201    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e296857_2009_48f2_9540_c8d94e468c88.slice/crio-00994e2ad2f5aa52f295b00d6d1a35e7a34bd653623f7b3bd11453c9763815c9 WatchSource:0}: Error finding container 00994e2ad2f5aa52f295b00d6d1a35e7a34bd653623f7b3bd11453c9763815c9: Status 404 returned error can't find the container with id 00994e2ad2f5aa52f295b00d6d1a35e7a34bd653623f7b3bd11453c9763815c9
Jul 28 02:21:27.063854 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:27.063811    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"7e296857-2009-48f2-9540-c8d94e468c88","Type":"ContainerStarted","Data":"4dcbf52cbc3166dacf122288dbed786bfb6cd8334dbe35b6ec99b03503dd0a86"}
Jul 28 02:21:27.064066 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:27.063879    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"7e296857-2009-48f2-9540-c8d94e468c88","Type":"ContainerStarted","Data":"00994e2ad2f5aa52f295b00d6d1a35e7a34bd653623f7b3bd11453c9763815c9"}
Jul 28 02:21:27.088026 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:27.087970    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" podStartSLOduration=1.087953448 podStartE2EDuration="1.087953448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:21:27.086865213 +0000 UTC m=+3330.664322831" watchObservedRunningTime="2026-07-28 02:21:27.087953448 +0000 UTC m=+3330.665411065"
Jul 28 02:21:27.491811 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:27.491720    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:36.491118 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:36.491084    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:36.504152 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:36.504128    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:21:37.120978 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:21:37.120946    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:24:45.752568 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:24:45.752543    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schegjj2w_553127d1-2dd7-4c00-a50f-abdd6b190186/main/0.log"
Jul 28 02:25:24.531969 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.531931    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 28 02:25:24.535477 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.535460    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:25:24.540911 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.540873    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 28 02:25:24.540911 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.540883    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 28 02:25:24.541103 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.540926    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-5jrnk\""
Jul 28 02:25:24.541199 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.541182    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 28 02:25:24.541241 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.541204    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-m98nw\""
Jul 28 02:25:24.546779 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.546754    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 28 02:25:24.576762 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.576737    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndddm\" (UniqueName: \"kubernetes.io/projected/1579c6d1-9645-4acd-b180-a7e0becfe665-kube-api-access-ndddm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1579c6d1-9645-4acd-b180-a7e0becfe665\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:25:24.576893 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.576782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1579c6d1-9645-4acd-b180-a7e0becfe665-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1579c6d1-9645-4acd-b180-a7e0becfe665\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:25:24.576893 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.576842    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1579c6d1-9645-4acd-b180-a7e0becfe665-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1579c6d1-9645-4acd-b180-a7e0becfe665\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:25:24.576893 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.576879    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1579c6d1-9645-4acd-b180-a7e0becfe665-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1579c6d1-9645-4acd-b180-a7e0becfe665\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:25:24.577002 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.576898    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1579c6d1-9645-4acd-b180-a7e0becfe665-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1579c6d1-9645-4acd-b180-a7e0becfe665\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:25:24.577002 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.576947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1579c6d1-9645-4acd-b180-a7e0becfe665-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1579c6d1-9645-4acd-b180-a7e0becfe665\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:25:24.677465 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.677429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1579c6d1-9645-4acd-b180-a7e0becfe665-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1579c6d1-9645-4acd-b180-a7e0becfe665\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:25:24.677666 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.677482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndddm\" (UniqueName: \"kubernetes.io/projected/1579c6d1-9645-4acd-b180-a7e0becfe665-kube-api-access-ndddm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1579c6d1-9645-4acd-b180-a7e0becfe665\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:25:24.677666 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.677517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1579c6d1-9645-4acd-b180-a7e0becfe665-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1579c6d1-9645-4acd-b180-a7e0becfe665\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:25:24.677666 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.677538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1579c6d1-9645-4acd-b180-a7e0becfe665-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1579c6d1-9645-4acd-b180-a7e0becfe665\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:25:24.677666 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.677554    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1579c6d1-9645-4acd-b180-a7e0becfe665-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1579c6d1-9645-4acd-b180-a7e0becfe665\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:25:24.677666 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.677568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1579c6d1-9645-4acd-b180-a7e0becfe665-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1579c6d1-9645-4acd-b180-a7e0becfe665\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:25:24.677930 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.677908    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1579c6d1-9645-4acd-b180-a7e0becfe665-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1579c6d1-9645-4acd-b180-a7e0becfe665\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:25:24.677990 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.677951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1579c6d1-9645-4acd-b180-a7e0becfe665-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1579c6d1-9645-4acd-b180-a7e0becfe665\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:25:24.678048 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.678022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1579c6d1-9645-4acd-b180-a7e0becfe665-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1579c6d1-9645-4acd-b180-a7e0becfe665\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:25:24.679823 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.679800    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1579c6d1-9645-4acd-b180-a7e0becfe665-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1579c6d1-9645-4acd-b180-a7e0becfe665\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:25:24.679945 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.679927    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1579c6d1-9645-4acd-b180-a7e0becfe665-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1579c6d1-9645-4acd-b180-a7e0becfe665\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:25:24.685845 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.685827    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndddm\" (UniqueName: \"kubernetes.io/projected/1579c6d1-9645-4acd-b180-a7e0becfe665-kube-api-access-ndddm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"1579c6d1-9645-4acd-b180-a7e0becfe665\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:25:24.746736 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.746698    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6"]
Jul 28 02:25:24.750050 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.750028    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6"
Jul 28 02:25:24.753234 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.753214    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-b2mns\""
Jul 28 02:25:24.758604 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.758583    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6"]
Jul 28 02:25:24.778553 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.778531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7zmr\" (UniqueName: \"kubernetes.io/projected/7b91a615-cf63-457c-bfcb-74c5828e6f48-kube-api-access-h7zmr\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6\" (UID: \"7b91a615-cf63-457c-bfcb-74c5828e6f48\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6"
Jul 28 02:25:24.778666 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.778581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b91a615-cf63-457c-bfcb-74c5828e6f48-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6\" (UID: \"7b91a615-cf63-457c-bfcb-74c5828e6f48\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6"
Jul 28 02:25:24.879703 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.879597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b91a615-cf63-457c-bfcb-74c5828e6f48-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6\" (UID: \"7b91a615-cf63-457c-bfcb-74c5828e6f48\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6"
Jul 28 02:25:24.879703 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.879680    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7zmr\" (UniqueName: \"kubernetes.io/projected/7b91a615-cf63-457c-bfcb-74c5828e6f48-kube-api-access-h7zmr\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6\" (UID: \"7b91a615-cf63-457c-bfcb-74c5828e6f48\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6"
Jul 28 02:25:24.882214 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.882184    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b91a615-cf63-457c-bfcb-74c5828e6f48-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6\" (UID: \"7b91a615-cf63-457c-bfcb-74c5828e6f48\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6"
Jul 28 02:25:24.888940 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:24.888915    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7zmr\" (UniqueName: \"kubernetes.io/projected/7b91a615-cf63-457c-bfcb-74c5828e6f48-kube-api-access-h7zmr\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6\" (UID: \"7b91a615-cf63-457c-bfcb-74c5828e6f48\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6"
Jul 28 02:25:25.061297 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:25.061264    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6"
Jul 28 02:25:25.187440 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:25.187407    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6"]
Jul 28 02:25:25.192021 ip-10-0-139-81 kubenswrapper[2578]: W0728 02:25:25.191996    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b91a615_cf63_457c_bfcb_74c5828e6f48.slice/crio-89b7617bd0954afd2f5a02f82e3df1a483db5eabe61b294da592728747b8ee61 WatchSource:0}: Error finding container 89b7617bd0954afd2f5a02f82e3df1a483db5eabe61b294da592728747b8ee61: Status 404 returned error can't find the container with id 89b7617bd0954afd2f5a02f82e3df1a483db5eabe61b294da592728747b8ee61
Jul 28 02:25:26.078206 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:26.078169    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6" event={"ID":"7b91a615-cf63-457c-bfcb-74c5828e6f48","Type":"ContainerStarted","Data":"bbacfb1cd95ec144ce5020a460830f005be5e746a70a60fb0eac43a86a5fb230"}
Jul 28 02:25:26.078626 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:26.078228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6" event={"ID":"7b91a615-cf63-457c-bfcb-74c5828e6f48","Type":"ContainerStarted","Data":"89b7617bd0954afd2f5a02f82e3df1a483db5eabe61b294da592728747b8ee61"}
Jul 28 02:25:26.078626 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:26.078275    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6"
Jul 28 02:25:26.099525 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:26.099483    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6" podStartSLOduration=2.099470443 podStartE2EDuration="2.099470443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:25:26.098284898 +0000 UTC m=+3569.675742530" watchObservedRunningTime="2026-07-28 02:25:26.099470443 +0000 UTC m=+3569.676928060"
Jul 28 02:25:57.084844 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:25:57.084809    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6"
Jul 28 02:27:27.545377 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:27:27.545273    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 02:27:27.614811 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:27:27.614779    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:29:30.616053 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:29:30.615951    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 02:29:31.143580 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:29:31.143532    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:31:34.144289 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:31:34.144249    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 02:31:34.668538 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:31:34.668499    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:33:37.669835 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:33:37.669739    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 02:33:38.181595 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:33:38.181567    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:35:41.182687 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:35:41.182630    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 02:35:41.712266 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:35:41.712238    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:36:22.249417 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.249314    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 28 02:36:22.258765 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:36:22.258729    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 02:36:22.258765 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:36:22.258765 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 02:36:22.258765 ip-10-0-139-81 kubenswrapper[2578]:  > containerID="4dcbf52cbc3166dacf122288dbed786bfb6cd8334dbe35b6ec99b03503dd0a86" cmd=["/bin/sleep","15"]
Jul 28 02:36:22.259105 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.258780    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" message="" err=<
Jul 28 02:36:22.259105 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:36:22.259105 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 02:36:22.259105 ip-10-0-139-81 kubenswrapper[2578]:  >
Jul 28 02:36:22.259105 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:36:22.258793    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 02:36:22.259105 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:36:22.259105 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 02:36:22.259105 ip-10-0-139-81 kubenswrapper[2578]:  > pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" podUID="7e296857-2009-48f2-9540-c8d94e468c88" containerName="main" containerID="cri-o://4dcbf52cbc3166dacf122288dbed786bfb6cd8334dbe35b6ec99b03503dd0a86"
Jul 28 02:36:22.259105 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.258841    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" podUID="7e296857-2009-48f2-9540-c8d94e468c88" containerName="main" containerID="cri-o://4dcbf52cbc3166dacf122288dbed786bfb6cd8334dbe35b6ec99b03503dd0a86" gracePeriod=60
Jul 28 02:36:22.606698 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.606672    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:36:22.660560 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.660525    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-model-cache\") pod \"7e296857-2009-48f2-9540-c8d94e468c88\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") "
Jul 28 02:36:22.660734 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.660582    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-tmp-dir\") pod \"7e296857-2009-48f2-9540-c8d94e468c88\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") "
Jul 28 02:36:22.660734 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.660610    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e296857-2009-48f2-9540-c8d94e468c88-tls-certs\") pod \"7e296857-2009-48f2-9540-c8d94e468c88\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") "
Jul 28 02:36:22.660734 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.660677    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7w8k\" (UniqueName: \"kubernetes.io/projected/7e296857-2009-48f2-9540-c8d94e468c88-kube-api-access-w7w8k\") pod \"7e296857-2009-48f2-9540-c8d94e468c88\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") "
Jul 28 02:36:22.660734 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.660717    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-dshm\") pod \"7e296857-2009-48f2-9540-c8d94e468c88\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") "
Jul 28 02:36:22.660920 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.660762    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-home\") pod \"7e296857-2009-48f2-9540-c8d94e468c88\" (UID: \"7e296857-2009-48f2-9540-c8d94e468c88\") "
Jul 28 02:36:22.660920 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.660862    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e296857-2009-48f2-9540-c8d94e468c88" (UID: "7e296857-2009-48f2-9540-c8d94e468c88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:36:22.660920 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.660880    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e296857-2009-48f2-9540-c8d94e468c88" (UID: "7e296857-2009-48f2-9540-c8d94e468c88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:36:22.661104 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.661069    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-home" (OuterVolumeSpecName: "home") pod "7e296857-2009-48f2-9540-c8d94e468c88" (UID: "7e296857-2009-48f2-9540-c8d94e468c88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:36:22.661171 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.661110    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:36:22.661171 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.661127    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:36:22.663002 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.662977    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-dshm" (OuterVolumeSpecName: "dshm") pod "7e296857-2009-48f2-9540-c8d94e468c88" (UID: "7e296857-2009-48f2-9540-c8d94e468c88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:36:22.663295 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.663275    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e296857-2009-48f2-9540-c8d94e468c88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e296857-2009-48f2-9540-c8d94e468c88" (UID: "7e296857-2009-48f2-9540-c8d94e468c88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:36:22.663388 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.663299    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e296857-2009-48f2-9540-c8d94e468c88-kube-api-access-w7w8k" (OuterVolumeSpecName: "kube-api-access-w7w8k") pod "7e296857-2009-48f2-9540-c8d94e468c88" (UID: "7e296857-2009-48f2-9540-c8d94e468c88"). InnerVolumeSpecName "kube-api-access-w7w8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:36:22.761716 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.761677    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e296857-2009-48f2-9540-c8d94e468c88-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:36:22.761716 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.761712    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7w8k\" (UniqueName: \"kubernetes.io/projected/7e296857-2009-48f2-9540-c8d94e468c88-kube-api-access-w7w8k\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:36:22.761716 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.761724    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:36:22.761970 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.761737    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e296857-2009-48f2-9540-c8d94e468c88-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:36:22.889103 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.889013    2578 generic.go:356] "Generic (PLEG): container finished" podID="7e296857-2009-48f2-9540-c8d94e468c88" containerID="4dcbf52cbc3166dacf122288dbed786bfb6cd8334dbe35b6ec99b03503dd0a86" exitCode=0
Jul 28 02:36:22.889103 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.889054    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 28 02:36:22.889301 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.889101    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"7e296857-2009-48f2-9540-c8d94e468c88","Type":"ContainerDied","Data":"4dcbf52cbc3166dacf122288dbed786bfb6cd8334dbe35b6ec99b03503dd0a86"}
Jul 28 02:36:22.889301 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.889155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"7e296857-2009-48f2-9540-c8d94e468c88","Type":"ContainerDied","Data":"00994e2ad2f5aa52f295b00d6d1a35e7a34bd653623f7b3bd11453c9763815c9"}
Jul 28 02:36:22.889301 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.889179    2578 scope.go:117] "RemoveContainer" containerID="4dcbf52cbc3166dacf122288dbed786bfb6cd8334dbe35b6ec99b03503dd0a86"
Jul 28 02:36:22.898120 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.898100    2578 scope.go:117] "RemoveContainer" containerID="4dcbf52cbc3166dacf122288dbed786bfb6cd8334dbe35b6ec99b03503dd0a86"
Jul 28 02:36:22.898379 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:36:22.898357    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dcbf52cbc3166dacf122288dbed786bfb6cd8334dbe35b6ec99b03503dd0a86\": container with ID starting with 4dcbf52cbc3166dacf122288dbed786bfb6cd8334dbe35b6ec99b03503dd0a86 not found: ID does not exist" containerID="4dcbf52cbc3166dacf122288dbed786bfb6cd8334dbe35b6ec99b03503dd0a86"
Jul 28 02:36:22.898484 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.898383    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dcbf52cbc3166dacf122288dbed786bfb6cd8334dbe35b6ec99b03503dd0a86"} err="failed to get container status \"4dcbf52cbc3166dacf122288dbed786bfb6cd8334dbe35b6ec99b03503dd0a86\": rpc error: code = NotFound desc = could not find container \"4dcbf52cbc3166dacf122288dbed786bfb6cd8334dbe35b6ec99b03503dd0a86\": container with ID starting with 4dcbf52cbc3166dacf122288dbed786bfb6cd8334dbe35b6ec99b03503dd0a86 not found: ID does not exist"
Jul 28 02:36:22.910615 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.910594    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 28 02:36:22.916271 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.916250    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 28 02:36:22.985410 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:36:22.985382    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e296857-2009-48f2-9540-c8d94e468c88" path="/var/lib/kubelet/pods/7e296857-2009-48f2-9540-c8d94e468c88/volumes"
Jul 28 02:37:09.878516 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:09.878483    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 02:37:09.878919 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:09.878837    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e296857-2009-48f2-9540-c8d94e468c88" containerName="main"
Jul 28 02:37:09.878919 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:09.878849    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e296857-2009-48f2-9540-c8d94e468c88" containerName="main"
Jul 28 02:37:09.878995 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:09.878927    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e296857-2009-48f2-9540-c8d94e468c88" containerName="main"
Jul 28 02:37:09.885742 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:09.885714    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:09.890566 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:09.890542    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"default-dockercfg-t4mf7\""
Jul 28 02:37:09.890874 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:09.890595    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-self-signed-certs\""
Jul 28 02:37:09.891012 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:09.890612    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"openshift-service-ca.crt\""
Jul 28 02:37:09.891012 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:09.890611    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"kube-root-ca.crt\""
Jul 28 02:37:09.891012 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:09.890931    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-mn-dockercfg-dlfz9\""
Jul 28 02:37:09.892571 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:09.892551    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 02:37:09.949837 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:09.949809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:09.949837 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:09.949838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:09.949992 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:09.949867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:09.949992 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:09.949963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:09.950064 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:09.950010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd7nq\" (UniqueName: \"kubernetes.io/projected/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-kube-api-access-xd7nq\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:09.950099 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:09.950080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:10.051157 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:10.051127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd7nq\" (UniqueName: \"kubernetes.io/projected/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-kube-api-access-xd7nq\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:10.051307 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:10.051175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:10.051307 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:10.051205    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:10.051307 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:10.051223    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:10.051307 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:10.051248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:10.051557 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:10.051409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:10.051649 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:10.051621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:10.051713 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:10.051681    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:10.051756 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:10.051721    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:10.053571 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:10.053550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:10.053683 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:10.053665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:10.063138 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:10.063117    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd7nq\" (UniqueName: \"kubernetes.io/projected/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-kube-api-access-xd7nq\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:10.199169 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:10.199100    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:37:10.328977 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:10.328948    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 02:37:10.332637 ip-10-0-139-81 kubenswrapper[2578]: W0728 02:37:10.332610    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fdeda3_09a1_4857_8fd4_50e9f74004ca.slice/crio-ae3a8ea815881d2cbc533b7f70739f2d52d00d285231bf15904b15b1488a76cb WatchSource:0}: Error finding container ae3a8ea815881d2cbc533b7f70739f2d52d00d285231bf15904b15b1488a76cb: Status 404 returned error can't find the container with id ae3a8ea815881d2cbc533b7f70739f2d52d00d285231bf15904b15b1488a76cb
Jul 28 02:37:11.094844 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:11.094811    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"f1fdeda3-09a1-4857-8fd4-50e9f74004ca","Type":"ContainerStarted","Data":"b50424d7b8136af7ae9f3299fa8549466a642e84d56921fbf8328b4e5b6506c8"}
Jul 28 02:37:11.095210 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:11.094873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"f1fdeda3-09a1-4857-8fd4-50e9f74004ca","Type":"ContainerStarted","Data":"ae3a8ea815881d2cbc533b7f70739f2d52d00d285231bf15904b15b1488a76cb"}
Jul 28 02:37:11.113010 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:11.112970    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podStartSLOduration=2.112956951 podStartE2EDuration="2.112956951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:37:11.111267674 +0000 UTC m=+4274.688725447" watchObservedRunningTime="2026-07-28 02:37:11.112956951 +0000 UTC m=+4274.690414568"
Jul 28 02:37:44.712917 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:37:44.712872    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 02:37:45.248525 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:37:45.248493    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:39:48.249502 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:39:48.249453    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 02:39:48.777889 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:39:48.777854    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:40:09.580505 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:40:09.580476    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7c3633d1_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schelxtl6_7b91a615-cf63-457c-bfcb-74c5828e6f48/main/0.log"
Jul 28 02:41:01.918677 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:01.918592    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"]
Jul 28 02:41:01.921947 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:01.921924    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:01.924692 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:01.924654    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 28 02:41:01.924933 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:01.924908    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 28 02:41:01.925048 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:01.924978    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 28 02:41:01.925177 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:01.925077    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-r9p95\""
Jul 28 02:41:01.925177 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:01.925145    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 28 02:41:01.945182 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:01.945153    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"]
Jul 28 02:41:02.035099 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.035065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/758e8c48-0a16-4937-8c3c-f0c560825541-tls-certs\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.035320 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.035136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-kserve-provision-location\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.035320 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.035179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-dshm\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.035320 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.035217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-tmp-dir\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.035320 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.035251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlhpr\" (UniqueName: \"kubernetes.io/projected/758e8c48-0a16-4937-8c3c-f0c560825541-kube-api-access-zlhpr\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.035320 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.035288    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-home\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.035320 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.035309    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-model-cache\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.135813 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.135781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/758e8c48-0a16-4937-8c3c-f0c560825541-tls-certs\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.136009 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.135838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-kserve-provision-location\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.136009 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.135870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-dshm\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.136009 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.135896    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-tmp-dir\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.136009 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.135917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlhpr\" (UniqueName: \"kubernetes.io/projected/758e8c48-0a16-4937-8c3c-f0c560825541-kube-api-access-zlhpr\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.136009 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.135939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-home\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.136293 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.136056    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-model-cache\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.136371 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.136324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-kserve-provision-location\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.136428 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.136369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-tmp-dir\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.136428 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.136409    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-home\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.136496 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.136483    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-model-cache\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.138257 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.138236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-dshm\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.138443 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.138427    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/758e8c48-0a16-4937-8c3c-f0c560825541-tls-certs\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.143544 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.143524    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlhpr\" (UniqueName: \"kubernetes.io/projected/758e8c48-0a16-4937-8c3c-f0c560825541-kube-api-access-zlhpr\") pod \"stop-feature-test-kserve-57d79c4c49-9k4h2\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.233689 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.233584    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:02.365578 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:02.365471    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"]
Jul 28 02:41:02.371165 ip-10-0-139-81 kubenswrapper[2578]: W0728 02:41:02.371138    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758e8c48_0a16_4937_8c3c_f0c560825541.slice/crio-0abaa89f5ab030ed08928501dc9f176c02f4a6390af0433f160e3bc1406c36fd WatchSource:0}: Error finding container 0abaa89f5ab030ed08928501dc9f176c02f4a6390af0433f160e3bc1406c36fd: Status 404 returned error can't find the container with id 0abaa89f5ab030ed08928501dc9f176c02f4a6390af0433f160e3bc1406c36fd
Jul 28 02:41:03.091743 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:03.091707    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" event={"ID":"758e8c48-0a16-4937-8c3c-f0c560825541","Type":"ContainerStarted","Data":"413a0c8c346c53fad344ef43e38e214e107b29106c44b1bee656f11951c0d721"}
Jul 28 02:41:03.092243 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:03.091761    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" event={"ID":"758e8c48-0a16-4937-8c3c-f0c560825541","Type":"ContainerStarted","Data":"0abaa89f5ab030ed08928501dc9f176c02f4a6390af0433f160e3bc1406c36fd"}
Jul 28 02:41:07.111079 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:07.110994    2578 generic.go:356] "Generic (PLEG): container finished" podID="758e8c48-0a16-4937-8c3c-f0c560825541" containerID="413a0c8c346c53fad344ef43e38e214e107b29106c44b1bee656f11951c0d721" exitCode=0
Jul 28 02:41:07.111079 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:07.111067    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" event={"ID":"758e8c48-0a16-4937-8c3c-f0c560825541","Type":"ContainerDied","Data":"413a0c8c346c53fad344ef43e38e214e107b29106c44b1bee656f11951c0d721"}
Jul 28 02:41:08.119728 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:08.119685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" event={"ID":"758e8c48-0a16-4937-8c3c-f0c560825541","Type":"ContainerStarted","Data":"bb0eda9eff3d2c17539bad0d3f8e908a23b8c67c5895aa79a91344d1f1390c25"}
Jul 28 02:41:08.142284 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:08.142228    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" podStartSLOduration=2.749420846 podStartE2EDuration="7.142207819s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.392786973s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:41:08.140876398 +0000 UTC m=+4511.718334018" watchObservedRunningTime="2026-07-28 02:41:08.142207819 +0000 UTC m=+4511.719665434"
Jul 28 02:41:08.233725 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:08.233690    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:12.234315 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:12.234216    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:41:12.235798 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:12.235768    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" podUID="758e8c48-0a16-4937-8c3c-f0c560825541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 28 02:41:22.234802 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:22.234744    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" podUID="758e8c48-0a16-4937-8c3c-f0c560825541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 28 02:41:32.234581 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:32.234540    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" podUID="758e8c48-0a16-4937-8c3c-f0c560825541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 28 02:41:42.234628 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:42.234580    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" podUID="758e8c48-0a16-4937-8c3c-f0c560825541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 28 02:41:51.778731 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:41:51.778681    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 02:41:52.234772 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:52.234666    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" podUID="758e8c48-0a16-4937-8c3c-f0c560825541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 28 02:41:52.326868 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:41:52.326834    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:42:02.234892 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:42:02.234850    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" podUID="758e8c48-0a16-4937-8c3c-f0c560825541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 28 02:42:12.234586 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:42:12.234494    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" podUID="758e8c48-0a16-4937-8c3c-f0c560825541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 28 02:42:22.234444 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:42:22.234323    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" podUID="758e8c48-0a16-4937-8c3c-f0c560825541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 28 02:42:32.234399 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:42:32.234348    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" podUID="758e8c48-0a16-4937-8c3c-f0c560825541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 28 02:42:42.234792 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:42:42.234747    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" podUID="758e8c48-0a16-4937-8c3c-f0c560825541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 28 02:42:52.234867 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:42:52.234823    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" podUID="758e8c48-0a16-4937-8c3c-f0c560825541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 28 02:43:02.243907 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:43:02.243877    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:43:02.251915 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:43:02.251891    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:43:44.816832 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:43:44.816796    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"]
Jul 28 02:43:55.328119 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:43:55.328072    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 02:43:55.889251 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:43:55.889219    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:43:59.839093 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:43:59.839054    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" podUID="758e8c48-0a16-4937-8c3c-f0c560825541" containerName="main" containerID="cri-o://bb0eda9eff3d2c17539bad0d3f8e908a23b8c67c5895aa79a91344d1f1390c25" gracePeriod=45
Jul 28 02:44:00.245049 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.244953    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" podUID="758e8c48-0a16-4937-8c3c-f0c560825541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 28 02:44:00.594870 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.594843    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:44:00.705168 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.705119    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-dshm\") pod \"758e8c48-0a16-4937-8c3c-f0c560825541\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") "
Jul 28 02:44:00.705389 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.705263    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-model-cache\") pod \"758e8c48-0a16-4937-8c3c-f0c560825541\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") "
Jul 28 02:44:00.705389 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.705299    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlhpr\" (UniqueName: \"kubernetes.io/projected/758e8c48-0a16-4937-8c3c-f0c560825541-kube-api-access-zlhpr\") pod \"758e8c48-0a16-4937-8c3c-f0c560825541\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") "
Jul 28 02:44:00.705389 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.705369    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/758e8c48-0a16-4937-8c3c-f0c560825541-tls-certs\") pod \"758e8c48-0a16-4937-8c3c-f0c560825541\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") "
Jul 28 02:44:00.705566 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.705394    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-tmp-dir\") pod \"758e8c48-0a16-4937-8c3c-f0c560825541\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") "
Jul 28 02:44:00.705566 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.705434    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-home\") pod \"758e8c48-0a16-4937-8c3c-f0c560825541\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") "
Jul 28 02:44:00.705566 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.705525    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-kserve-provision-location\") pod \"758e8c48-0a16-4937-8c3c-f0c560825541\" (UID: \"758e8c48-0a16-4937-8c3c-f0c560825541\") "
Jul 28 02:44:00.705725 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.705605    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-model-cache" (OuterVolumeSpecName: "model-cache") pod "758e8c48-0a16-4937-8c3c-f0c560825541" (UID: "758e8c48-0a16-4937-8c3c-f0c560825541"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:44:00.705907 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.705868    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:44:00.706113 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.706088    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-home" (OuterVolumeSpecName: "home") pod "758e8c48-0a16-4937-8c3c-f0c560825541" (UID: "758e8c48-0a16-4937-8c3c-f0c560825541"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:44:00.707720 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.707695    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/758e8c48-0a16-4937-8c3c-f0c560825541-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "758e8c48-0a16-4937-8c3c-f0c560825541" (UID: "758e8c48-0a16-4937-8c3c-f0c560825541"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:44:00.707911 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.707883    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-dshm" (OuterVolumeSpecName: "dshm") pod "758e8c48-0a16-4937-8c3c-f0c560825541" (UID: "758e8c48-0a16-4937-8c3c-f0c560825541"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:44:00.707995 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.707960    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758e8c48-0a16-4937-8c3c-f0c560825541-kube-api-access-zlhpr" (OuterVolumeSpecName: "kube-api-access-zlhpr") pod "758e8c48-0a16-4937-8c3c-f0c560825541" (UID: "758e8c48-0a16-4937-8c3c-f0c560825541"). InnerVolumeSpecName "kube-api-access-zlhpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:44:00.723222 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.723182    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "758e8c48-0a16-4937-8c3c-f0c560825541" (UID: "758e8c48-0a16-4937-8c3c-f0c560825541"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:44:00.774249 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.774203    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "758e8c48-0a16-4937-8c3c-f0c560825541" (UID: "758e8c48-0a16-4937-8c3c-f0c560825541"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:44:00.806842 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.806802    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlhpr\" (UniqueName: \"kubernetes.io/projected/758e8c48-0a16-4937-8c3c-f0c560825541-kube-api-access-zlhpr\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:44:00.806842 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.806834    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/758e8c48-0a16-4937-8c3c-f0c560825541-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:44:00.806842 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.806844    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:44:00.807096 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.806852    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:44:00.807096 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.806862    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-kserve-provision-location\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:44:00.807096 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.806870    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/758e8c48-0a16-4937-8c3c-f0c560825541-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:44:00.917315 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.917280    2578 generic.go:356] "Generic (PLEG): container finished" podID="758e8c48-0a16-4937-8c3c-f0c560825541" containerID="bb0eda9eff3d2c17539bad0d3f8e908a23b8c67c5895aa79a91344d1f1390c25" exitCode=0
Jul 28 02:44:00.917745 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.917353    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"
Jul 28 02:44:00.917745 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.917352    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" event={"ID":"758e8c48-0a16-4937-8c3c-f0c560825541","Type":"ContainerDied","Data":"bb0eda9eff3d2c17539bad0d3f8e908a23b8c67c5895aa79a91344d1f1390c25"}
Jul 28 02:44:00.917745 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.917419    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2" event={"ID":"758e8c48-0a16-4937-8c3c-f0c560825541","Type":"ContainerDied","Data":"0abaa89f5ab030ed08928501dc9f176c02f4a6390af0433f160e3bc1406c36fd"}
Jul 28 02:44:00.917745 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.917450    2578 scope.go:117] "RemoveContainer" containerID="bb0eda9eff3d2c17539bad0d3f8e908a23b8c67c5895aa79a91344d1f1390c25"
Jul 28 02:44:00.926750 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.926725    2578 scope.go:117] "RemoveContainer" containerID="413a0c8c346c53fad344ef43e38e214e107b29106c44b1bee656f11951c0d721"
Jul 28 02:44:00.936597 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.936573    2578 scope.go:117] "RemoveContainer" containerID="bb0eda9eff3d2c17539bad0d3f8e908a23b8c67c5895aa79a91344d1f1390c25"
Jul 28 02:44:00.936917 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:44:00.936895    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0eda9eff3d2c17539bad0d3f8e908a23b8c67c5895aa79a91344d1f1390c25\": container with ID starting with bb0eda9eff3d2c17539bad0d3f8e908a23b8c67c5895aa79a91344d1f1390c25 not found: ID does not exist" containerID="bb0eda9eff3d2c17539bad0d3f8e908a23b8c67c5895aa79a91344d1f1390c25"
Jul 28 02:44:00.937001 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.936931    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0eda9eff3d2c17539bad0d3f8e908a23b8c67c5895aa79a91344d1f1390c25"} err="failed to get container status \"bb0eda9eff3d2c17539bad0d3f8e908a23b8c67c5895aa79a91344d1f1390c25\": rpc error: code = NotFound desc = could not find container \"bb0eda9eff3d2c17539bad0d3f8e908a23b8c67c5895aa79a91344d1f1390c25\": container with ID starting with bb0eda9eff3d2c17539bad0d3f8e908a23b8c67c5895aa79a91344d1f1390c25 not found: ID does not exist"
Jul 28 02:44:00.937001 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.936958    2578 scope.go:117] "RemoveContainer" containerID="413a0c8c346c53fad344ef43e38e214e107b29106c44b1bee656f11951c0d721"
Jul 28 02:44:00.937224 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:44:00.937207    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"413a0c8c346c53fad344ef43e38e214e107b29106c44b1bee656f11951c0d721\": container with ID starting with 413a0c8c346c53fad344ef43e38e214e107b29106c44b1bee656f11951c0d721 not found: ID does not exist" containerID="413a0c8c346c53fad344ef43e38e214e107b29106c44b1bee656f11951c0d721"
Jul 28 02:44:00.937266 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.937231    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"413a0c8c346c53fad344ef43e38e214e107b29106c44b1bee656f11951c0d721"} err="failed to get container status \"413a0c8c346c53fad344ef43e38e214e107b29106c44b1bee656f11951c0d721\": rpc error: code = NotFound desc = could not find container \"413a0c8c346c53fad344ef43e38e214e107b29106c44b1bee656f11951c0d721\": container with ID starting with 413a0c8c346c53fad344ef43e38e214e107b29106c44b1bee656f11951c0d721 not found: ID does not exist"
Jul 28 02:44:00.940861 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.940836    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"]
Jul 28 02:44:00.945750 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.945728    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-9k4h2"]
Jul 28 02:44:00.986935 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:00.986897    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="758e8c48-0a16-4937-8c3c-f0c560825541" path="/var/lib/kubelet/pods/758e8c48-0a16-4937-8c3c-f0c560825541/volumes"
Jul 28 02:44:08.019965 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.019926    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"]
Jul 28 02:44:08.020410 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.020358    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="758e8c48-0a16-4937-8c3c-f0c560825541" containerName="main"
Jul 28 02:44:08.020410 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.020372    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="758e8c48-0a16-4937-8c3c-f0c560825541" containerName="main"
Jul 28 02:44:08.020410 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.020384    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="758e8c48-0a16-4937-8c3c-f0c560825541" containerName="storage-initializer"
Jul 28 02:44:08.020410 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.020394    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="758e8c48-0a16-4937-8c3c-f0c560825541" containerName="storage-initializer"
Jul 28 02:44:08.020550 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.020450    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="758e8c48-0a16-4937-8c3c-f0c560825541" containerName="main"
Jul 28 02:44:08.025324 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.025303    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.028161 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.028137    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-r9p95\""
Jul 28 02:44:08.028292 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.028159    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 28 02:44:08.028292 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.028210    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 28 02:44:08.028292 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.028259    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 28 02:44:08.028292 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.028288    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 28 02:44:08.034168 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.034147    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"]
Jul 28 02:44:08.173686 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.173650    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-kserve-provision-location\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.173686 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.173694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-dshm\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.173929 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.173783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-home\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.173929 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.173863    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwzzh\" (UniqueName: \"kubernetes.io/projected/51c3c4da-e909-422b-89fb-7af76aa1cd37-kube-api-access-qwzzh\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.173929 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.173888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-tmp-dir\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.173929 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.173905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c3c4da-e909-422b-89fb-7af76aa1cd37-tls-certs\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.174056 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.173945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-model-cache\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.274952 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.274865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-home\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.274952 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.274934    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwzzh\" (UniqueName: \"kubernetes.io/projected/51c3c4da-e909-422b-89fb-7af76aa1cd37-kube-api-access-qwzzh\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.275180 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.274961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-tmp-dir\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.275180 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.274981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c3c4da-e909-422b-89fb-7af76aa1cd37-tls-certs\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.275180 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.275004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-model-cache\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.275180 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.275047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-kserve-provision-location\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.275180 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.275082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-dshm\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.275457 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.275368    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-home\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.275457 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.275398    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-tmp-dir\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.275665 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.275642    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-model-cache\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.275750 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.275698    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-kserve-provision-location\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.277928 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.277903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-dshm\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.278176 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.278160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c3c4da-e909-422b-89fb-7af76aa1cd37-tls-certs\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.298623 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.298589    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwzzh\" (UniqueName: \"kubernetes.io/projected/51c3c4da-e909-422b-89fb-7af76aa1cd37-kube-api-access-qwzzh\") pod \"stop-feature-test-kserve-57d79c4c49-95qcz\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.337358 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.337297    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:08.494839 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.494807    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"]
Jul 28 02:44:08.494839 ip-10-0-139-81 kubenswrapper[2578]: W0728 02:44:08.494826    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c3c4da_e909_422b_89fb_7af76aa1cd37.slice/crio-62d5d4ab9f53dfda75c7cdfe3ffe0df48c5b579b6da07e1313783bebc894c531 WatchSource:0}: Error finding container 62d5d4ab9f53dfda75c7cdfe3ffe0df48c5b579b6da07e1313783bebc894c531: Status 404 returned error can't find the container with id 62d5d4ab9f53dfda75c7cdfe3ffe0df48c5b579b6da07e1313783bebc894c531
Jul 28 02:44:08.959770 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.959666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" event={"ID":"51c3c4da-e909-422b-89fb-7af76aa1cd37","Type":"ContainerStarted","Data":"43221bd3192d335fa6dea196cb7b32642fce8001d642b6e04e68159718571aaf"}
Jul 28 02:44:08.959770 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:08.959741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" event={"ID":"51c3c4da-e909-422b-89fb-7af76aa1cd37","Type":"ContainerStarted","Data":"62d5d4ab9f53dfda75c7cdfe3ffe0df48c5b579b6da07e1313783bebc894c531"}
Jul 28 02:44:18.005451 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:18.005418    2578 generic.go:356] "Generic (PLEG): container finished" podID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerID="43221bd3192d335fa6dea196cb7b32642fce8001d642b6e04e68159718571aaf" exitCode=0
Jul 28 02:44:18.005926 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:18.005501    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" event={"ID":"51c3c4da-e909-422b-89fb-7af76aa1cd37","Type":"ContainerDied","Data":"43221bd3192d335fa6dea196cb7b32642fce8001d642b6e04e68159718571aaf"}
Jul 28 02:44:19.015778 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:19.015730    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" event={"ID":"51c3c4da-e909-422b-89fb-7af76aa1cd37","Type":"ContainerStarted","Data":"67e79e5eb911847b44371373dac4c86236b3b55a2b1e32d0bd8cbc3db26cd580"}
Jul 28 02:44:19.040935 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:19.040858    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" podStartSLOduration=3.277645448 podStartE2EDuration="12.040833297s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.763187849s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:44:19.037212599 +0000 UTC m=+4702.614670257" watchObservedRunningTime="2026-07-28 02:44:19.040833297 +0000 UTC m=+4702.618290915"
Jul 28 02:44:19.338483 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:19.338400    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:28.338364 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:28.338310    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:44:28.339925 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:28.339896    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" podUID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 28 02:44:38.338750 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:38.338702    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" podUID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 28 02:44:48.338523 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:48.338481    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" podUID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 28 02:44:58.338104 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:44:58.338064    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" podUID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 28 02:45:08.337945 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:45:08.337885    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" podUID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 28 02:45:18.338546 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:45:18.338456    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" podUID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 28 02:45:28.338203 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:45:28.338149    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" podUID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 28 02:45:38.338670 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:45:38.338618    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" podUID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 28 02:45:48.338185 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:45:48.338143    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" podUID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 28 02:45:58.337932 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:45:58.337887    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" podUID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 28 02:45:58.890117 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:45:58.890074    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 02:45:59.495272 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:45:59.495241    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:46:08.338561 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:08.338517    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" podUID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 28 02:46:18.347450 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:18.347415    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:46:18.354970 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:18.354940    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:46:38.102354 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:38.095477    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"]
Jul 28 02:46:53.126242 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:53.126126    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" podUID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerName="main" containerID="cri-o://67e79e5eb911847b44371373dac4c86236b3b55a2b1e32d0bd8cbc3db26cd580" gracePeriod=45
Jul 28 02:46:53.349062 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:53.349011    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" podUID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 28 02:46:53.886953 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:53.886923    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:46:53.952006 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:53.951917    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c3c4da-e909-422b-89fb-7af76aa1cd37-tls-certs\") pod \"51c3c4da-e909-422b-89fb-7af76aa1cd37\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") "
Jul 28 02:46:53.952006 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:53.951956    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwzzh\" (UniqueName: \"kubernetes.io/projected/51c3c4da-e909-422b-89fb-7af76aa1cd37-kube-api-access-qwzzh\") pod \"51c3c4da-e909-422b-89fb-7af76aa1cd37\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") "
Jul 28 02:46:53.952248 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:53.952013    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-home\") pod \"51c3c4da-e909-422b-89fb-7af76aa1cd37\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") "
Jul 28 02:46:53.952248 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:53.952031    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-tmp-dir\") pod \"51c3c4da-e909-422b-89fb-7af76aa1cd37\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") "
Jul 28 02:46:53.952248 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:53.952053    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-dshm\") pod \"51c3c4da-e909-422b-89fb-7af76aa1cd37\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") "
Jul 28 02:46:53.952248 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:53.952082    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-kserve-provision-location\") pod \"51c3c4da-e909-422b-89fb-7af76aa1cd37\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") "
Jul 28 02:46:53.952248 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:53.952103    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-model-cache\") pod \"51c3c4da-e909-422b-89fb-7af76aa1cd37\" (UID: \"51c3c4da-e909-422b-89fb-7af76aa1cd37\") "
Jul 28 02:46:53.952562 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:53.952541    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-model-cache" (OuterVolumeSpecName: "model-cache") pod "51c3c4da-e909-422b-89fb-7af76aa1cd37" (UID: "51c3c4da-e909-422b-89fb-7af76aa1cd37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:46:53.952870 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:53.952841    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-home" (OuterVolumeSpecName: "home") pod "51c3c4da-e909-422b-89fb-7af76aa1cd37" (UID: "51c3c4da-e909-422b-89fb-7af76aa1cd37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:46:53.954417 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:53.954382    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-dshm" (OuterVolumeSpecName: "dshm") pod "51c3c4da-e909-422b-89fb-7af76aa1cd37" (UID: "51c3c4da-e909-422b-89fb-7af76aa1cd37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:46:53.954526 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:53.954413    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51c3c4da-e909-422b-89fb-7af76aa1cd37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51c3c4da-e909-422b-89fb-7af76aa1cd37" (UID: "51c3c4da-e909-422b-89fb-7af76aa1cd37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:46:53.954591 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:53.954578    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c3c4da-e909-422b-89fb-7af76aa1cd37-kube-api-access-qwzzh" (OuterVolumeSpecName: "kube-api-access-qwzzh") pod "51c3c4da-e909-422b-89fb-7af76aa1cd37" (UID: "51c3c4da-e909-422b-89fb-7af76aa1cd37"). InnerVolumeSpecName "kube-api-access-qwzzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:46:53.964583 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:53.964554    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51c3c4da-e909-422b-89fb-7af76aa1cd37" (UID: "51c3c4da-e909-422b-89fb-7af76aa1cd37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:46:54.019088 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.019042    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51c3c4da-e909-422b-89fb-7af76aa1cd37" (UID: "51c3c4da-e909-422b-89fb-7af76aa1cd37"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:46:54.053484 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.053445    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:46:54.053484 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.053478    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:46:54.053484 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.053487    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:46:54.053484 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.053495    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-kserve-provision-location\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:46:54.053759 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.053504    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c3c4da-e909-422b-89fb-7af76aa1cd37-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:46:54.053759 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.053514    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c3c4da-e909-422b-89fb-7af76aa1cd37-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:46:54.053759 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.053522    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwzzh\" (UniqueName: \"kubernetes.io/projected/51c3c4da-e909-422b-89fb-7af76aa1cd37-kube-api-access-qwzzh\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:46:54.740474 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.740438    2578 generic.go:356] "Generic (PLEG): container finished" podID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerID="67e79e5eb911847b44371373dac4c86236b3b55a2b1e32d0bd8cbc3db26cd580" exitCode=0
Jul 28 02:46:54.740474 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.740479    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" event={"ID":"51c3c4da-e909-422b-89fb-7af76aa1cd37","Type":"ContainerDied","Data":"67e79e5eb911847b44371373dac4c86236b3b55a2b1e32d0bd8cbc3db26cd580"}
Jul 28 02:46:54.741059 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.740497    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"
Jul 28 02:46:54.741059 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.740530    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz" event={"ID":"51c3c4da-e909-422b-89fb-7af76aa1cd37","Type":"ContainerDied","Data":"62d5d4ab9f53dfda75c7cdfe3ffe0df48c5b579b6da07e1313783bebc894c531"}
Jul 28 02:46:54.741059 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.740561    2578 scope.go:117] "RemoveContainer" containerID="67e79e5eb911847b44371373dac4c86236b3b55a2b1e32d0bd8cbc3db26cd580"
Jul 28 02:46:54.749256 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.749239    2578 scope.go:117] "RemoveContainer" containerID="43221bd3192d335fa6dea196cb7b32642fce8001d642b6e04e68159718571aaf"
Jul 28 02:46:54.757430 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.757409    2578 scope.go:117] "RemoveContainer" containerID="67e79e5eb911847b44371373dac4c86236b3b55a2b1e32d0bd8cbc3db26cd580"
Jul 28 02:46:54.757689 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:46:54.757669    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e79e5eb911847b44371373dac4c86236b3b55a2b1e32d0bd8cbc3db26cd580\": container with ID starting with 67e79e5eb911847b44371373dac4c86236b3b55a2b1e32d0bd8cbc3db26cd580 not found: ID does not exist" containerID="67e79e5eb911847b44371373dac4c86236b3b55a2b1e32d0bd8cbc3db26cd580"
Jul 28 02:46:54.757745 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.757697    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e79e5eb911847b44371373dac4c86236b3b55a2b1e32d0bd8cbc3db26cd580"} err="failed to get container status \"67e79e5eb911847b44371373dac4c86236b3b55a2b1e32d0bd8cbc3db26cd580\": rpc error: code = NotFound desc = could not find container \"67e79e5eb911847b44371373dac4c86236b3b55a2b1e32d0bd8cbc3db26cd580\": container with ID starting with 67e79e5eb911847b44371373dac4c86236b3b55a2b1e32d0bd8cbc3db26cd580 not found: ID does not exist"
Jul 28 02:46:54.757745 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.757713    2578 scope.go:117] "RemoveContainer" containerID="43221bd3192d335fa6dea196cb7b32642fce8001d642b6e04e68159718571aaf"
Jul 28 02:46:54.757962 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:46:54.757940    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43221bd3192d335fa6dea196cb7b32642fce8001d642b6e04e68159718571aaf\": container with ID starting with 43221bd3192d335fa6dea196cb7b32642fce8001d642b6e04e68159718571aaf not found: ID does not exist" containerID="43221bd3192d335fa6dea196cb7b32642fce8001d642b6e04e68159718571aaf"
Jul 28 02:46:54.758003 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.757970    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43221bd3192d335fa6dea196cb7b32642fce8001d642b6e04e68159718571aaf"} err="failed to get container status \"43221bd3192d335fa6dea196cb7b32642fce8001d642b6e04e68159718571aaf\": rpc error: code = NotFound desc = could not find container \"43221bd3192d335fa6dea196cb7b32642fce8001d642b6e04e68159718571aaf\": container with ID starting with 43221bd3192d335fa6dea196cb7b32642fce8001d642b6e04e68159718571aaf not found: ID does not exist"
Jul 28 02:46:54.763780 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.763759    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"]
Jul 28 02:46:54.767756 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.767738    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-95qcz"]
Jul 28 02:46:54.984964 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:46:54.984933    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51c3c4da-e909-422b-89fb-7af76aa1cd37" path="/var/lib/kubelet/pods/51c3c4da-e909-422b-89fb-7af76aa1cd37/volumes"
Jul 28 02:47:21.032230 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.032194    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"]
Jul 28 02:47:21.032703 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.032534    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerName="storage-initializer"
Jul 28 02:47:21.032703 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.032546    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerName="storage-initializer"
Jul 28 02:47:21.032703 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.032555    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerName="main"
Jul 28 02:47:21.032703 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.032561    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerName="main"
Jul 28 02:47:21.032703 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.032625    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="51c3c4da-e909-422b-89fb-7af76aa1cd37" containerName="main"
Jul 28 02:47:21.035957 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.035939    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.038688 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.038664    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Jul 28 02:47:21.038968 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.038951    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 28 02:47:21.039067 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.038971    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:47:21.039067 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.039013    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Jul 28 02:47:21.039183 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.039086    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Jul 28 02:47:21.039246 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.039217    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-npd4n\""
Jul 28 02:47:21.050618 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.050592    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"]
Jul 28 02:47:21.187439 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.187407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-home\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.187439 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.187440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeddb555-caf0-419b-8c2b-39c3a3477190-tls-certs\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.187737 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.187517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-dshm\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.187737 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.187542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9ldw\" (UniqueName: \"kubernetes.io/projected/eeddb555-caf0-419b-8c2b-39c3a3477190-kube-api-access-g9ldw\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.187737 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.187610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eeddb555-caf0-419b-8c2b-39c3a3477190-cabundle-cert\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.187737 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.187674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-model-cache\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.187737 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.187700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-tmp-dir\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.187737 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.187731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.288557 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.288472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-dshm\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.288557 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.288514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9ldw\" (UniqueName: \"kubernetes.io/projected/eeddb555-caf0-419b-8c2b-39c3a3477190-kube-api-access-g9ldw\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.288746 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.288636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eeddb555-caf0-419b-8c2b-39c3a3477190-cabundle-cert\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.288746 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.288679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-model-cache\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.288746 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.288704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-tmp-dir\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.288746 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.288738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.288927 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.288768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-home\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.288927 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.288790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeddb555-caf0-419b-8c2b-39c3a3477190-tls-certs\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.289095 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.289066    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-model-cache\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.289180 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.289112    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.289180 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.289165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-home\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.289375 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.289353    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-tmp-dir\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.289464 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.289389    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eeddb555-caf0-419b-8c2b-39c3a3477190-cabundle-cert\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.290966 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.290936    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-dshm\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.291204 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.291186    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeddb555-caf0-419b-8c2b-39c3a3477190-tls-certs\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.296466 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.296445    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9ldw\" (UniqueName: \"kubernetes.io/projected/eeddb555-caf0-419b-8c2b-39c3a3477190-kube-api-access-g9ldw\") pod \"lora-single-adapter-test-kserve-9f49df58-frl5d\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.346206 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.346166    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:21.484976 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.484888    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"]
Jul 28 02:47:21.487305 ip-10-0-139-81 kubenswrapper[2578]: W0728 02:47:21.487271    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeddb555_caf0_419b_8c2b_39c3a3477190.slice/crio-9a03c142baaa538f4e71aa9b8a6e884a6faeb0a7a71b8af31808a7dadf53efd0 WatchSource:0}: Error finding container 9a03c142baaa538f4e71aa9b8a6e884a6faeb0a7a71b8af31808a7dadf53efd0: Status 404 returned error can't find the container with id 9a03c142baaa538f4e71aa9b8a6e884a6faeb0a7a71b8af31808a7dadf53efd0
Jul 28 02:47:21.862697 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.862657    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d" event={"ID":"eeddb555-caf0-419b-8c2b-39c3a3477190","Type":"ContainerStarted","Data":"0dcd4eea3160cfcd6f084ec5013de3913bd57e4d25510be38bf8db4534e3311a"}
Jul 28 02:47:21.862890 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:21.862715    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d" event={"ID":"eeddb555-caf0-419b-8c2b-39c3a3477190","Type":"ContainerStarted","Data":"9a03c142baaa538f4e71aa9b8a6e884a6faeb0a7a71b8af31808a7dadf53efd0"}
Jul 28 02:47:26.888610 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:26.888572    2578 generic.go:356] "Generic (PLEG): container finished" podID="eeddb555-caf0-419b-8c2b-39c3a3477190" containerID="0dcd4eea3160cfcd6f084ec5013de3913bd57e4d25510be38bf8db4534e3311a" exitCode=0
Jul 28 02:47:26.889060 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:26.888644    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d" event={"ID":"eeddb555-caf0-419b-8c2b-39c3a3477190","Type":"ContainerDied","Data":"0dcd4eea3160cfcd6f084ec5013de3913bd57e4d25510be38bf8db4534e3311a"}
Jul 28 02:47:27.894718 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:27.894685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d" event={"ID":"eeddb555-caf0-419b-8c2b-39c3a3477190","Type":"ContainerStarted","Data":"733b04cf401cabf7622a2b7d85425cab73ca591ed746bcc0917b4f44b003ac4c"}
Jul 28 02:47:27.917462 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:27.917400    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d" podStartSLOduration=2.190241816 podStartE2EDuration="6.917383607s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.727141791s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:47:27.916232634 +0000 UTC m=+4891.493690249" watchObservedRunningTime="2026-07-28 02:47:27.917383607 +0000 UTC m=+4891.494841217"
Jul 28 02:47:28.346797 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:28.346745    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:31.347301 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:31.347255    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:47:31.348943 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:31.348905    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d" podUID="eeddb555-caf0-419b-8c2b-39c3a3477190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 28 02:47:41.346907 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:41.346855    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d" podUID="eeddb555-caf0-419b-8c2b-39c3a3477190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 28 02:47:51.347377 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:47:51.347308    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d" podUID="eeddb555-caf0-419b-8c2b-39c3a3477190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 28 02:48:01.347285 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:48:01.347241    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d" podUID="eeddb555-caf0-419b-8c2b-39c3a3477190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 28 02:48:02.496393 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:48:02.496354    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 02:48:03.062057 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:48:03.062026    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:48:11.347640 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:48:11.347600    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d" podUID="eeddb555-caf0-419b-8c2b-39c3a3477190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 28 02:48:21.346991 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:48:21.346893    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d" podUID="eeddb555-caf0-419b-8c2b-39c3a3477190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 28 02:48:31.347547 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:48:31.347504    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d" podUID="eeddb555-caf0-419b-8c2b-39c3a3477190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 28 02:48:41.347355 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:48:41.347300    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d" podUID="eeddb555-caf0-419b-8c2b-39c3a3477190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 28 02:48:51.361689 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:48:51.361657    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:48:51.369967 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:48:51.369938    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:49:36.052045 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:49:36.052014    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-9f49df58-frl5d_eeddb555-caf0-419b-8c2b-39c3a3477190/storage-initializer/0.log"
Jul 28 02:49:36.064779 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:49:36.064736    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-9f49df58-frl5d_eeddb555-caf0-419b-8c2b-39c3a3477190/main/0.log"
Jul 28 02:49:54.969024 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:49:54.968930    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"]
Jul 28 02:50:06.063348 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:50:06.063293    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 02:50:06.696008 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:06.695976    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:50:09.991692 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:09.991632    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d" podUID="eeddb555-caf0-419b-8c2b-39c3a3477190" containerName="main" containerID="cri-o://733b04cf401cabf7622a2b7d85425cab73ca591ed746bcc0917b4f44b003ac4c" gracePeriod=45
Jul 28 02:50:10.363109 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.363061    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d" podUID="eeddb555-caf0-419b-8c2b-39c3a3477190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 28 02:50:10.636944 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.636919    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:50:10.713793 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.713765    2578 generic.go:356] "Generic (PLEG): container finished" podID="eeddb555-caf0-419b-8c2b-39c3a3477190" containerID="733b04cf401cabf7622a2b7d85425cab73ca591ed746bcc0917b4f44b003ac4c" exitCode=0
Jul 28 02:50:10.713930 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.713813    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"
Jul 28 02:50:10.713930 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.713855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d" event={"ID":"eeddb555-caf0-419b-8c2b-39c3a3477190","Type":"ContainerDied","Data":"733b04cf401cabf7622a2b7d85425cab73ca591ed746bcc0917b4f44b003ac4c"}
Jul 28 02:50:10.713930 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.713908    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d" event={"ID":"eeddb555-caf0-419b-8c2b-39c3a3477190","Type":"ContainerDied","Data":"9a03c142baaa538f4e71aa9b8a6e884a6faeb0a7a71b8af31808a7dadf53efd0"}
Jul 28 02:50:10.714060 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.713933    2578 scope.go:117] "RemoveContainer" containerID="733b04cf401cabf7622a2b7d85425cab73ca591ed746bcc0917b4f44b003ac4c"
Jul 28 02:50:10.721956 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.721935    2578 scope.go:117] "RemoveContainer" containerID="0dcd4eea3160cfcd6f084ec5013de3913bd57e4d25510be38bf8db4534e3311a"
Jul 28 02:50:10.731450 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.731430    2578 scope.go:117] "RemoveContainer" containerID="733b04cf401cabf7622a2b7d85425cab73ca591ed746bcc0917b4f44b003ac4c"
Jul 28 02:50:10.731730 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:50:10.731712    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733b04cf401cabf7622a2b7d85425cab73ca591ed746bcc0917b4f44b003ac4c\": container with ID starting with 733b04cf401cabf7622a2b7d85425cab73ca591ed746bcc0917b4f44b003ac4c not found: ID does not exist" containerID="733b04cf401cabf7622a2b7d85425cab73ca591ed746bcc0917b4f44b003ac4c"
Jul 28 02:50:10.731803 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.731738    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733b04cf401cabf7622a2b7d85425cab73ca591ed746bcc0917b4f44b003ac4c"} err="failed to get container status \"733b04cf401cabf7622a2b7d85425cab73ca591ed746bcc0917b4f44b003ac4c\": rpc error: code = NotFound desc = could not find container \"733b04cf401cabf7622a2b7d85425cab73ca591ed746bcc0917b4f44b003ac4c\": container with ID starting with 733b04cf401cabf7622a2b7d85425cab73ca591ed746bcc0917b4f44b003ac4c not found: ID does not exist"
Jul 28 02:50:10.731803 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.731760    2578 scope.go:117] "RemoveContainer" containerID="0dcd4eea3160cfcd6f084ec5013de3913bd57e4d25510be38bf8db4534e3311a"
Jul 28 02:50:10.732009 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:50:10.731991    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dcd4eea3160cfcd6f084ec5013de3913bd57e4d25510be38bf8db4534e3311a\": container with ID starting with 0dcd4eea3160cfcd6f084ec5013de3913bd57e4d25510be38bf8db4534e3311a not found: ID does not exist" containerID="0dcd4eea3160cfcd6f084ec5013de3913bd57e4d25510be38bf8db4534e3311a"
Jul 28 02:50:10.732059 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.732016    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dcd4eea3160cfcd6f084ec5013de3913bd57e4d25510be38bf8db4534e3311a"} err="failed to get container status \"0dcd4eea3160cfcd6f084ec5013de3913bd57e4d25510be38bf8db4534e3311a\": rpc error: code = NotFound desc = could not find container \"0dcd4eea3160cfcd6f084ec5013de3913bd57e4d25510be38bf8db4534e3311a\": container with ID starting with 0dcd4eea3160cfcd6f084ec5013de3913bd57e4d25510be38bf8db4534e3311a not found: ID does not exist"
Jul 28 02:50:10.751854 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.751835    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-model-cache\") pod \"eeddb555-caf0-419b-8c2b-39c3a3477190\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") "
Jul 28 02:50:10.751939 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.751868    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-dshm\") pod \"eeddb555-caf0-419b-8c2b-39c3a3477190\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") "
Jul 28 02:50:10.751939 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.751908    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-home\") pod \"eeddb555-caf0-419b-8c2b-39c3a3477190\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") "
Jul 28 02:50:10.752113 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.751942    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-tmp-dir\") pod \"eeddb555-caf0-419b-8c2b-39c3a3477190\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") "
Jul 28 02:50:10.752113 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.751968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-kserve-provision-location\") pod \"eeddb555-caf0-419b-8c2b-39c3a3477190\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") "
Jul 28 02:50:10.752113 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.752035    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9ldw\" (UniqueName: \"kubernetes.io/projected/eeddb555-caf0-419b-8c2b-39c3a3477190-kube-api-access-g9ldw\") pod \"eeddb555-caf0-419b-8c2b-39c3a3477190\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") "
Jul 28 02:50:10.752277 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.752125    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-model-cache" (OuterVolumeSpecName: "model-cache") pod "eeddb555-caf0-419b-8c2b-39c3a3477190" (UID: "eeddb555-caf0-419b-8c2b-39c3a3477190"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:50:10.752277 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.752267    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeddb555-caf0-419b-8c2b-39c3a3477190-tls-certs\") pod \"eeddb555-caf0-419b-8c2b-39c3a3477190\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") "
Jul 28 02:50:10.752412 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.752304    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eeddb555-caf0-419b-8c2b-39c3a3477190-cabundle-cert\") pod \"eeddb555-caf0-419b-8c2b-39c3a3477190\" (UID: \"eeddb555-caf0-419b-8c2b-39c3a3477190\") "
Jul 28 02:50:10.752412 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.752354    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-home" (OuterVolumeSpecName: "home") pod "eeddb555-caf0-419b-8c2b-39c3a3477190" (UID: "eeddb555-caf0-419b-8c2b-39c3a3477190"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:50:10.752691 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.752661    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:50:10.752831 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.752810    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:50:10.753004 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.752961    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeddb555-caf0-419b-8c2b-39c3a3477190-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eeddb555-caf0-419b-8c2b-39c3a3477190" (UID: "eeddb555-caf0-419b-8c2b-39c3a3477190"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:50:10.754687 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.754664    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeddb555-caf0-419b-8c2b-39c3a3477190-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eeddb555-caf0-419b-8c2b-39c3a3477190" (UID: "eeddb555-caf0-419b-8c2b-39c3a3477190"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:50:10.754840 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.754820    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-dshm" (OuterVolumeSpecName: "dshm") pod "eeddb555-caf0-419b-8c2b-39c3a3477190" (UID: "eeddb555-caf0-419b-8c2b-39c3a3477190"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:50:10.754933 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.754916    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeddb555-caf0-419b-8c2b-39c3a3477190-kube-api-access-g9ldw" (OuterVolumeSpecName: "kube-api-access-g9ldw") pod "eeddb555-caf0-419b-8c2b-39c3a3477190" (UID: "eeddb555-caf0-419b-8c2b-39c3a3477190"). InnerVolumeSpecName "kube-api-access-g9ldw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:50:10.768130 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.768101    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eeddb555-caf0-419b-8c2b-39c3a3477190" (UID: "eeddb555-caf0-419b-8c2b-39c3a3477190"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:50:10.817817 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.817782    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eeddb555-caf0-419b-8c2b-39c3a3477190" (UID: "eeddb555-caf0-419b-8c2b-39c3a3477190"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:50:10.853536 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.853509    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:50:10.853536 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.853532    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:50:10.853677 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.853548    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeddb555-caf0-419b-8c2b-39c3a3477190-kserve-provision-location\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:50:10.853677 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.853561    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9ldw\" (UniqueName: \"kubernetes.io/projected/eeddb555-caf0-419b-8c2b-39c3a3477190-kube-api-access-g9ldw\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:50:10.853677 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.853574    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeddb555-caf0-419b-8c2b-39c3a3477190-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:50:10.853677 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:10.853587    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eeddb555-caf0-419b-8c2b-39c3a3477190-cabundle-cert\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:50:11.030523 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:11.030489    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"]
Jul 28 02:50:11.034003 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:11.033977    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-frl5d"]
Jul 28 02:50:12.984876 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:12.984842    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeddb555-caf0-419b-8c2b-39c3a3477190" path="/var/lib/kubelet/pods/eeddb555-caf0-419b-8c2b-39c3a3477190/volumes"
Jul 28 02:50:27.826750 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.826717    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"]
Jul 28 02:50:27.827191 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.827049    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeddb555-caf0-419b-8c2b-39c3a3477190" containerName="storage-initializer"
Jul 28 02:50:27.827191 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.827060    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeddb555-caf0-419b-8c2b-39c3a3477190" containerName="storage-initializer"
Jul 28 02:50:27.827191 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.827079    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeddb555-caf0-419b-8c2b-39c3a3477190" containerName="main"
Jul 28 02:50:27.827191 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.827084    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeddb555-caf0-419b-8c2b-39c3a3477190" containerName="main"
Jul 28 02:50:27.827191 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.827139    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeddb555-caf0-419b-8c2b-39c3a3477190" containerName="main"
Jul 28 02:50:27.830064 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.830043    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:27.833179 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.833156    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Jul 28 02:50:27.833588 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.833570    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Jul 28 02:50:27.833769 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.833750    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 02:50:27.833894 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.833877    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Jul 28 02:50:27.834033 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.833980    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 28 02:50:27.834097 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.834054    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-n6lqm\""
Jul 28 02:50:27.840938 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.840917    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"]
Jul 28 02:50:27.996246 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.996212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-model-cache\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:27.996246 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.996252    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-dshm\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:27.996507 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.996273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:27.996507 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.996402    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5npq\" (UniqueName: \"kubernetes.io/projected/5b411384-f0b2-4c7f-be80-3f479258fe3b-kube-api-access-t5npq\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:27.996507 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.996429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-home\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:27.996507 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.996450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:27.996656 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.996544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b411384-f0b2-4c7f-be80-3f479258fe3b-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:27.996656 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:27.996568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b411384-f0b2-4c7f-be80-3f479258fe3b-tls-certs\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:28.097516 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.097406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-dshm\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:28.097516 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.097466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:28.097764 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.097594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5npq\" (UniqueName: \"kubernetes.io/projected/5b411384-f0b2-4c7f-be80-3f479258fe3b-kube-api-access-t5npq\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:28.097764 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.097641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-home\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:28.097764 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.097686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:28.097938 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.097858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b411384-f0b2-4c7f-be80-3f479258fe3b-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:28.097938 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.097898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b411384-f0b2-4c7f-be80-3f479258fe3b-tls-certs\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:28.097938 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.097916    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:28.098100 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.097961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:28.098100 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.097965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-model-cache\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:28.098100 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.098006    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-home\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:28.098261 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.098196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-model-cache\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:28.098469 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.098449    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b411384-f0b2-4c7f-be80-3f479258fe3b-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:28.100107 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.100076    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-dshm\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:28.100300 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.100283    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b411384-f0b2-4c7f-be80-3f479258fe3b-tls-certs\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:28.105631 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.105606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5npq\" (UniqueName: \"kubernetes.io/projected/5b411384-f0b2-4c7f-be80-3f479258fe3b-kube-api-access-t5npq\") pod \"lora-multiple-adapters-test-kserve-9ffb494ff-tk42q\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:28.141558 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.141526    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:28.282678 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.282651    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"]
Jul 28 02:50:28.284715 ip-10-0-139-81 kubenswrapper[2578]: W0728 02:50:28.284689    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b411384_f0b2_4c7f_be80_3f479258fe3b.slice/crio-fb870c11da658b4028b5025c0a7e631dd9134212bba9bf4e51f67ce602f26c9e WatchSource:0}: Error finding container fb870c11da658b4028b5025c0a7e631dd9134212bba9bf4e51f67ce602f26c9e: Status 404 returned error can't find the container with id fb870c11da658b4028b5025c0a7e631dd9134212bba9bf4e51f67ce602f26c9e
Jul 28 02:50:28.796881 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.796841    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q" event={"ID":"5b411384-f0b2-4c7f-be80-3f479258fe3b","Type":"ContainerStarted","Data":"824c2f64de3532307767ee1489dc5ff5cd35c395dcd59cb7b9ed2eb5f66aa519"}
Jul 28 02:50:28.797083 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:28.796895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q" event={"ID":"5b411384-f0b2-4c7f-be80-3f479258fe3b","Type":"ContainerStarted","Data":"fb870c11da658b4028b5025c0a7e631dd9134212bba9bf4e51f67ce602f26c9e"}
Jul 28 02:50:33.826113 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:33.826073    2578 generic.go:356] "Generic (PLEG): container finished" podID="5b411384-f0b2-4c7f-be80-3f479258fe3b" containerID="824c2f64de3532307767ee1489dc5ff5cd35c395dcd59cb7b9ed2eb5f66aa519" exitCode=0
Jul 28 02:50:33.826551 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:33.826138    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q" event={"ID":"5b411384-f0b2-4c7f-be80-3f479258fe3b","Type":"ContainerDied","Data":"824c2f64de3532307767ee1489dc5ff5cd35c395dcd59cb7b9ed2eb5f66aa519"}
Jul 28 02:50:34.833118 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:34.833081    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q" event={"ID":"5b411384-f0b2-4c7f-be80-3f479258fe3b","Type":"ContainerStarted","Data":"b10000394c1b9db5d290c0f859e314ec0ef6ec040e692d44eedfb21446ae93eb"}
Jul 28 02:50:34.859349 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:34.859278    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q" podStartSLOduration=3.053202304 podStartE2EDuration="7.859260566s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.806058262s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:50:34.857676889 +0000 UTC m=+5078.435134507" watchObservedRunningTime="2026-07-28 02:50:34.859260566 +0000 UTC m=+5078.436718183"
Jul 28 02:50:35.142106 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:35.142008    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:38.142104 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:38.142070    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:50:38.143557 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:38.143526    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q" podUID="5b411384-f0b2-4c7f-be80-3f479258fe3b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 28 02:50:48.142217 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:48.142164    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q" podUID="5b411384-f0b2-4c7f-be80-3f479258fe3b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 28 02:50:58.142809 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:50:58.142760    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q" podUID="5b411384-f0b2-4c7f-be80-3f479258fe3b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 28 02:51:08.142740 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:51:08.142686    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q" podUID="5b411384-f0b2-4c7f-be80-3f479258fe3b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 28 02:51:18.142962 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:51:18.142909    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q" podUID="5b411384-f0b2-4c7f-be80-3f479258fe3b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 28 02:51:28.142843 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:51:28.142799    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q" podUID="5b411384-f0b2-4c7f-be80-3f479258fe3b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 28 02:51:38.142351 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:51:38.142296    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q" podUID="5b411384-f0b2-4c7f-be80-3f479258fe3b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 28 02:51:48.142626 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:51:48.142577    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q" podUID="5b411384-f0b2-4c7f-be80-3f479258fe3b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 28 02:51:58.152516 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:51:58.152483    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:51:58.160895 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:51:58.160865    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:52:07.404870 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.404838    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 02:52:07.413596 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:52:07.413560    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 02:52:07.413596 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:52:07.413596 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 02:52:07.413596 ip-10-0-139-81 kubenswrapper[2578]:  > containerID="b50424d7b8136af7ae9f3299fa8549466a642e84d56921fbf8328b4e5b6506c8" cmd=["/bin/sleep","15"]
Jul 28 02:52:07.413856 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.413610    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" message="" err=<
Jul 28 02:52:07.413856 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:52:07.413856 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 02:52:07.413856 ip-10-0-139-81 kubenswrapper[2578]:  >
Jul 28 02:52:07.413856 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:52:07.413622    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 02:52:07.413856 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 02:52:07.413856 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 02:52:07.413856 ip-10-0-139-81 kubenswrapper[2578]:  > pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podUID="f1fdeda3-09a1-4857-8fd4-50e9f74004ca" containerName="main" containerID="cri-o://b50424d7b8136af7ae9f3299fa8549466a642e84d56921fbf8328b4e5b6506c8"
Jul 28 02:52:07.413856 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.413689    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podUID="f1fdeda3-09a1-4857-8fd4-50e9f74004ca" containerName="main" containerID="cri-o://b50424d7b8136af7ae9f3299fa8549466a642e84d56921fbf8328b4e5b6506c8" gracePeriod=60
Jul 28 02:52:07.663169 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.663111    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:52:07.812412 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.812369    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-tls-certs\") pod \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") "
Jul 28 02:52:07.812623 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.812424    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-model-cache\") pod \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") "
Jul 28 02:52:07.812623 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.812465    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-tmp-dir\") pod \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") "
Jul 28 02:52:07.812623 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.812518    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd7nq\" (UniqueName: \"kubernetes.io/projected/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-kube-api-access-xd7nq\") pod \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") "
Jul 28 02:52:07.812623 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.812550    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-dshm\") pod \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") "
Jul 28 02:52:07.812623 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.812581    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-home\") pod \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\" (UID: \"f1fdeda3-09a1-4857-8fd4-50e9f74004ca\") "
Jul 28 02:52:07.812901 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.812775    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1fdeda3-09a1-4857-8fd4-50e9f74004ca" (UID: "f1fdeda3-09a1-4857-8fd4-50e9f74004ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:52:07.812901 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.812790    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1fdeda3-09a1-4857-8fd4-50e9f74004ca" (UID: "f1fdeda3-09a1-4857-8fd4-50e9f74004ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:52:07.812901 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.812879    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:52:07.812901 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.812901    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:52:07.813067 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.813012    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-home" (OuterVolumeSpecName: "home") pod "f1fdeda3-09a1-4857-8fd4-50e9f74004ca" (UID: "f1fdeda3-09a1-4857-8fd4-50e9f74004ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:52:07.815626 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.815598    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-kube-api-access-xd7nq" (OuterVolumeSpecName: "kube-api-access-xd7nq") pod "f1fdeda3-09a1-4857-8fd4-50e9f74004ca" (UID: "f1fdeda3-09a1-4857-8fd4-50e9f74004ca"). InnerVolumeSpecName "kube-api-access-xd7nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:52:07.815772 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.815659    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1fdeda3-09a1-4857-8fd4-50e9f74004ca" (UID: "f1fdeda3-09a1-4857-8fd4-50e9f74004ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:52:07.815772 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.815690    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-dshm" (OuterVolumeSpecName: "dshm") pod "f1fdeda3-09a1-4857-8fd4-50e9f74004ca" (UID: "f1fdeda3-09a1-4857-8fd4-50e9f74004ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:52:07.913980 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.913882    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:52:07.913980 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.913925    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd7nq\" (UniqueName: \"kubernetes.io/projected/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-kube-api-access-xd7nq\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:52:07.913980 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.913943    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:52:07.913980 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:07.913957    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1fdeda3-09a1-4857-8fd4-50e9f74004ca-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:52:08.294408 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:08.294375    2578 generic.go:356] "Generic (PLEG): container finished" podID="f1fdeda3-09a1-4857-8fd4-50e9f74004ca" containerID="b50424d7b8136af7ae9f3299fa8549466a642e84d56921fbf8328b4e5b6506c8" exitCode=0
Jul 28 02:52:08.294646 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:08.294424    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"f1fdeda3-09a1-4857-8fd4-50e9f74004ca","Type":"ContainerDied","Data":"b50424d7b8136af7ae9f3299fa8549466a642e84d56921fbf8328b4e5b6506c8"}
Jul 28 02:52:08.294646 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:08.294435    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 02:52:08.294646 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:08.294477    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"f1fdeda3-09a1-4857-8fd4-50e9f74004ca","Type":"ContainerDied","Data":"ae3a8ea815881d2cbc533b7f70739f2d52d00d285231bf15904b15b1488a76cb"}
Jul 28 02:52:08.294646 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:08.294502    2578 scope.go:117] "RemoveContainer" containerID="b50424d7b8136af7ae9f3299fa8549466a642e84d56921fbf8328b4e5b6506c8"
Jul 28 02:52:08.304317 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:08.304295    2578 scope.go:117] "RemoveContainer" containerID="b50424d7b8136af7ae9f3299fa8549466a642e84d56921fbf8328b4e5b6506c8"
Jul 28 02:52:08.304607 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:52:08.304587    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b50424d7b8136af7ae9f3299fa8549466a642e84d56921fbf8328b4e5b6506c8\": container with ID starting with b50424d7b8136af7ae9f3299fa8549466a642e84d56921fbf8328b4e5b6506c8 not found: ID does not exist" containerID="b50424d7b8136af7ae9f3299fa8549466a642e84d56921fbf8328b4e5b6506c8"
Jul 28 02:52:08.304665 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:08.304615    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b50424d7b8136af7ae9f3299fa8549466a642e84d56921fbf8328b4e5b6506c8"} err="failed to get container status \"b50424d7b8136af7ae9f3299fa8549466a642e84d56921fbf8328b4e5b6506c8\": rpc error: code = NotFound desc = could not find container \"b50424d7b8136af7ae9f3299fa8549466a642e84d56921fbf8328b4e5b6506c8\": container with ID starting with b50424d7b8136af7ae9f3299fa8549466a642e84d56921fbf8328b4e5b6506c8 not found: ID does not exist"
Jul 28 02:52:08.314905 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:08.314879    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 02:52:08.319554 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:08.319529    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 02:52:08.985467 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:08.985431    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1fdeda3-09a1-4857-8fd4-50e9f74004ca" path="/var/lib/kubelet/pods/f1fdeda3-09a1-4857-8fd4-50e9f74004ca/volumes"
Jul 28 02:52:09.697104 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:52:09.697066    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 02:52:10.304489 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:52:10.304463    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:53:18.324856 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:18.324824    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-9ffb494ff-tk42q_5b411384-f0b2-4c7f-be80-3f479258fe3b/storage-initializer/0.log"
Jul 28 02:53:18.333973 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:18.333951    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-9ffb494ff-tk42q_5b411384-f0b2-4c7f-be80-3f479258fe3b/main/0.log"
Jul 28 02:53:34.880594 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:34.880557    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"]
Jul 28 02:53:49.903109 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:49.903062    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q" podUID="5b411384-f0b2-4c7f-be80-3f479258fe3b" containerName="main" containerID="cri-o://b10000394c1b9db5d290c0f859e314ec0ef6ec040e692d44eedfb21446ae93eb" gracePeriod=45
Jul 28 02:53:50.153371 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.153258    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q" podUID="5b411384-f0b2-4c7f-be80-3f479258fe3b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 28 02:53:50.563523 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.563497    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:53:50.669467 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.669425    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-dshm\") pod \"5b411384-f0b2-4c7f-be80-3f479258fe3b\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") "
Jul 28 02:53:50.669467 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.669470    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5npq\" (UniqueName: \"kubernetes.io/projected/5b411384-f0b2-4c7f-be80-3f479258fe3b-kube-api-access-t5npq\") pod \"5b411384-f0b2-4c7f-be80-3f479258fe3b\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") "
Jul 28 02:53:50.669741 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.669499    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-model-cache\") pod \"5b411384-f0b2-4c7f-be80-3f479258fe3b\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") "
Jul 28 02:53:50.669741 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.669615    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-tmp-dir\") pod \"5b411384-f0b2-4c7f-be80-3f479258fe3b\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") "
Jul 28 02:53:50.669741 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.669647    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b411384-f0b2-4c7f-be80-3f479258fe3b-tls-certs\") pod \"5b411384-f0b2-4c7f-be80-3f479258fe3b\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") "
Jul 28 02:53:50.669741 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.669703    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-home\") pod \"5b411384-f0b2-4c7f-be80-3f479258fe3b\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") "
Jul 28 02:53:50.669741 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.669723    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-kserve-provision-location\") pod \"5b411384-f0b2-4c7f-be80-3f479258fe3b\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") "
Jul 28 02:53:50.670009 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.669751    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b411384-f0b2-4c7f-be80-3f479258fe3b-cabundle-cert\") pod \"5b411384-f0b2-4c7f-be80-3f479258fe3b\" (UID: \"5b411384-f0b2-4c7f-be80-3f479258fe3b\") "
Jul 28 02:53:50.670009 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.669785    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b411384-f0b2-4c7f-be80-3f479258fe3b" (UID: "5b411384-f0b2-4c7f-be80-3f479258fe3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:53:50.670130 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.670010    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:53:50.670130 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.670106    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-home" (OuterVolumeSpecName: "home") pod "5b411384-f0b2-4c7f-be80-3f479258fe3b" (UID: "5b411384-f0b2-4c7f-be80-3f479258fe3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:53:50.670231 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.670166    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b411384-f0b2-4c7f-be80-3f479258fe3b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5b411384-f0b2-4c7f-be80-3f479258fe3b" (UID: "5b411384-f0b2-4c7f-be80-3f479258fe3b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:53:50.672166 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.672098    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-dshm" (OuterVolumeSpecName: "dshm") pod "5b411384-f0b2-4c7f-be80-3f479258fe3b" (UID: "5b411384-f0b2-4c7f-be80-3f479258fe3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:53:50.672166 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.672100    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b411384-f0b2-4c7f-be80-3f479258fe3b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b411384-f0b2-4c7f-be80-3f479258fe3b" (UID: "5b411384-f0b2-4c7f-be80-3f479258fe3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:53:50.672316 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.672187    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b411384-f0b2-4c7f-be80-3f479258fe3b-kube-api-access-t5npq" (OuterVolumeSpecName: "kube-api-access-t5npq") pod "5b411384-f0b2-4c7f-be80-3f479258fe3b" (UID: "5b411384-f0b2-4c7f-be80-3f479258fe3b"). InnerVolumeSpecName "kube-api-access-t5npq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:53:50.687694 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.687658    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b411384-f0b2-4c7f-be80-3f479258fe3b" (UID: "5b411384-f0b2-4c7f-be80-3f479258fe3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:53:50.738132 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.738095    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b411384-f0b2-4c7f-be80-3f479258fe3b" (UID: "5b411384-f0b2-4c7f-be80-3f479258fe3b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:53:50.770851 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.770821    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:53:50.770851 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.770846    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b411384-f0b2-4c7f-be80-3f479258fe3b-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:53:50.770851 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.770855    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:53:50.771044 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.770865    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-kserve-provision-location\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:53:50.771044 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.770873    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b411384-f0b2-4c7f-be80-3f479258fe3b-cabundle-cert\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:53:50.771044 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.770881    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b411384-f0b2-4c7f-be80-3f479258fe3b-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:53:50.771044 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.770889    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5npq\" (UniqueName: \"kubernetes.io/projected/5b411384-f0b2-4c7f-be80-3f479258fe3b-kube-api-access-t5npq\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:53:50.820543 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.820508    2578 generic.go:356] "Generic (PLEG): container finished" podID="5b411384-f0b2-4c7f-be80-3f479258fe3b" containerID="b10000394c1b9db5d290c0f859e314ec0ef6ec040e692d44eedfb21446ae93eb" exitCode=0
Jul 28 02:53:50.820662 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.820558    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"
Jul 28 02:53:50.820662 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.820592    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q" event={"ID":"5b411384-f0b2-4c7f-be80-3f479258fe3b","Type":"ContainerDied","Data":"b10000394c1b9db5d290c0f859e314ec0ef6ec040e692d44eedfb21446ae93eb"}
Jul 28 02:53:50.820748 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.820667    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q" event={"ID":"5b411384-f0b2-4c7f-be80-3f479258fe3b","Type":"ContainerDied","Data":"fb870c11da658b4028b5025c0a7e631dd9134212bba9bf4e51f67ce602f26c9e"}
Jul 28 02:53:50.820748 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.820690    2578 scope.go:117] "RemoveContainer" containerID="b10000394c1b9db5d290c0f859e314ec0ef6ec040e692d44eedfb21446ae93eb"
Jul 28 02:53:50.829573 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.829554    2578 scope.go:117] "RemoveContainer" containerID="824c2f64de3532307767ee1489dc5ff5cd35c395dcd59cb7b9ed2eb5f66aa519"
Jul 28 02:53:50.837731 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.837706    2578 scope.go:117] "RemoveContainer" containerID="b10000394c1b9db5d290c0f859e314ec0ef6ec040e692d44eedfb21446ae93eb"
Jul 28 02:53:50.837975 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:53:50.837957    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10000394c1b9db5d290c0f859e314ec0ef6ec040e692d44eedfb21446ae93eb\": container with ID starting with b10000394c1b9db5d290c0f859e314ec0ef6ec040e692d44eedfb21446ae93eb not found: ID does not exist" containerID="b10000394c1b9db5d290c0f859e314ec0ef6ec040e692d44eedfb21446ae93eb"
Jul 28 02:53:50.838028 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.837983    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10000394c1b9db5d290c0f859e314ec0ef6ec040e692d44eedfb21446ae93eb"} err="failed to get container status \"b10000394c1b9db5d290c0f859e314ec0ef6ec040e692d44eedfb21446ae93eb\": rpc error: code = NotFound desc = could not find container \"b10000394c1b9db5d290c0f859e314ec0ef6ec040e692d44eedfb21446ae93eb\": container with ID starting with b10000394c1b9db5d290c0f859e314ec0ef6ec040e692d44eedfb21446ae93eb not found: ID does not exist"
Jul 28 02:53:50.838028 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.838001    2578 scope.go:117] "RemoveContainer" containerID="824c2f64de3532307767ee1489dc5ff5cd35c395dcd59cb7b9ed2eb5f66aa519"
Jul 28 02:53:50.838240 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:53:50.838223    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"824c2f64de3532307767ee1489dc5ff5cd35c395dcd59cb7b9ed2eb5f66aa519\": container with ID starting with 824c2f64de3532307767ee1489dc5ff5cd35c395dcd59cb7b9ed2eb5f66aa519 not found: ID does not exist" containerID="824c2f64de3532307767ee1489dc5ff5cd35c395dcd59cb7b9ed2eb5f66aa519"
Jul 28 02:53:50.838279 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.838246    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"824c2f64de3532307767ee1489dc5ff5cd35c395dcd59cb7b9ed2eb5f66aa519"} err="failed to get container status \"824c2f64de3532307767ee1489dc5ff5cd35c395dcd59cb7b9ed2eb5f66aa519\": rpc error: code = NotFound desc = could not find container \"824c2f64de3532307767ee1489dc5ff5cd35c395dcd59cb7b9ed2eb5f66aa519\": container with ID starting with 824c2f64de3532307767ee1489dc5ff5cd35c395dcd59cb7b9ed2eb5f66aa519 not found: ID does not exist"
Jul 28 02:53:50.842524 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.842502    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"]
Jul 28 02:53:50.849018 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.849000    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-9ffb494ff-tk42q"]
Jul 28 02:53:50.986473 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:53:50.986392    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b411384-f0b2-4c7f-be80-3f479258fe3b" path="/var/lib/kubelet/pods/5b411384-f0b2-4c7f-be80-3f479258fe3b/volumes"
Jul 28 02:54:13.305668 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:54:13.305621    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 02:54:13.916854 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:13.916822    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:54:41.948434 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:41.948352    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"]
Jul 28 02:54:41.948842 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:41.948669    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b411384-f0b2-4c7f-be80-3f479258fe3b" containerName="main"
Jul 28 02:54:41.948842 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:41.948680    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b411384-f0b2-4c7f-be80-3f479258fe3b" containerName="main"
Jul 28 02:54:41.948842 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:41.948698    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b411384-f0b2-4c7f-be80-3f479258fe3b" containerName="storage-initializer"
Jul 28 02:54:41.948842 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:41.948704    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b411384-f0b2-4c7f-be80-3f479258fe3b" containerName="storage-initializer"
Jul 28 02:54:41.948842 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:41.948711    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1fdeda3-09a1-4857-8fd4-50e9f74004ca" containerName="main"
Jul 28 02:54:41.948842 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:41.948717    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1fdeda3-09a1-4857-8fd4-50e9f74004ca" containerName="main"
Jul 28 02:54:41.948842 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:41.948773    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1fdeda3-09a1-4857-8fd4-50e9f74004ca" containerName="main"
Jul 28 02:54:41.948842 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:41.948781    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b411384-f0b2-4c7f-be80-3f479258fe3b" containerName="main"
Jul 28 02:54:41.951672 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:41.951655    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:41.954614 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:41.954593    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 28 02:54:41.954754 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:41.954719    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Jul 28 02:54:41.954801 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:41.954779    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-gkmdk\""
Jul 28 02:54:41.954846 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:41.954825    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Jul 28 02:54:41.954903 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:41.954890    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Jul 28 02:54:41.961000 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:41.960976    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"]
Jul 28 02:54:42.000826 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.000796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-tmp-dir\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.000990 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.000833    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-dshm\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.000990 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.000902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-model-cache\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.000990 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.000937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-kserve-provision-location\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.000990 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.000974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1965ea60-0e79-4e24-b214-9b4048eb6dbe-tls-certs\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.000990 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.000988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nfkv\" (UniqueName: \"kubernetes.io/projected/1965ea60-0e79-4e24-b214-9b4048eb6dbe-kube-api-access-7nfkv\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.001160 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.001045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-home\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.101897 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.101858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1965ea60-0e79-4e24-b214-9b4048eb6dbe-tls-certs\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.101897 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.101897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nfkv\" (UniqueName: \"kubernetes.io/projected/1965ea60-0e79-4e24-b214-9b4048eb6dbe-kube-api-access-7nfkv\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.102157 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.101934    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-home\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.102157 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.101979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-tmp-dir\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.102157 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.102010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-dshm\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.102157 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.102068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-model-cache\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.102157 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.102101    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-kserve-provision-location\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.102552 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.102527    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-tmp-dir\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.102552 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.102536    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-home\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.102701 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.102631    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-kserve-provision-location\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.102754 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.102694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-model-cache\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.104513 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.104491    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-dshm\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.104596 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.104579    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1965ea60-0e79-4e24-b214-9b4048eb6dbe-tls-certs\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.109632 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.109608    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nfkv\" (UniqueName: \"kubernetes.io/projected/1965ea60-0e79-4e24-b214-9b4048eb6dbe-kube-api-access-7nfkv\") pod \"tls-verification-test-kserve-b4c7b9b88-wrpzw\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.262683 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.262654    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:42.398835 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:42.398811    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"]
Jul 28 02:54:42.400983 ip-10-0-139-81 kubenswrapper[2578]: W0728 02:54:42.400951    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1965ea60_0e79_4e24_b214_9b4048eb6dbe.slice/crio-a436869f90a0bea523ab602b691789848acb5e3712653a5e221f817084253c6e WatchSource:0}: Error finding container a436869f90a0bea523ab602b691789848acb5e3712653a5e221f817084253c6e: Status 404 returned error can't find the container with id a436869f90a0bea523ab602b691789848acb5e3712653a5e221f817084253c6e
Jul 28 02:54:43.041018 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:43.040980    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw" event={"ID":"1965ea60-0e79-4e24-b214-9b4048eb6dbe","Type":"ContainerStarted","Data":"153a940b79f091cb46c457654ca40a06e70948195bcd7fbf02d3edb2f4957b5c"}
Jul 28 02:54:43.041666 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:43.041048    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw" event={"ID":"1965ea60-0e79-4e24-b214-9b4048eb6dbe","Type":"ContainerStarted","Data":"a436869f90a0bea523ab602b691789848acb5e3712653a5e221f817084253c6e"}
Jul 28 02:54:48.069730 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:48.069694    2578 generic.go:356] "Generic (PLEG): container finished" podID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" containerID="153a940b79f091cb46c457654ca40a06e70948195bcd7fbf02d3edb2f4957b5c" exitCode=0
Jul 28 02:54:48.070116 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:48.069767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw" event={"ID":"1965ea60-0e79-4e24-b214-9b4048eb6dbe","Type":"ContainerDied","Data":"153a940b79f091cb46c457654ca40a06e70948195bcd7fbf02d3edb2f4957b5c"}
Jul 28 02:54:49.076313 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:49.076277    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw" event={"ID":"1965ea60-0e79-4e24-b214-9b4048eb6dbe","Type":"ContainerStarted","Data":"8efca794defe05169b3c9869df6bcc4d1760a8ff814d52d26cb96d498fec2a22"}
Jul 28 02:54:49.098347 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:49.098264    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw" podStartSLOduration=3.510334048 podStartE2EDuration="8.098245854s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.587911806s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:54:49.096791461 +0000 UTC m=+5332.674249078" watchObservedRunningTime="2026-07-28 02:54:49.098245854 +0000 UTC m=+5332.675703471"
Jul 28 02:54:49.262962 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:49.262923    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:52.263397 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:52.263354    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:54:52.265220 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:54:52.265189    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw" podUID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 02:55:02.263775 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:55:02.263728    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw" podUID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 02:55:12.263984 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:55:12.263944    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw" podUID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 02:55:22.264167 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:55:22.264119    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw" podUID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 02:55:32.263185 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:55:32.263131    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw" podUID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 02:55:42.263750 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:55:42.263707    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw" podUID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 02:55:52.264183 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:55:52.264077    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw" podUID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 02:56:02.264217 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:56:02.264068    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw" podUID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 02:56:12.263745 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:56:12.263707    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw" podUID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 02:56:16.918001 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:56:16.917937    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 02:56:17.484900 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:56:17.484869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:56:22.263469 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:56:22.263415    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw" podUID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 02:56:32.263390 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:56:32.263322    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw" podUID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 28 02:56:42.273772 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:56:42.273729    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:56:42.281867 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:56:42.281840    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:57:07.340814 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:07.340773    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"]
Jul 28 02:57:22.364421 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:22.364317    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw" podUID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" containerName="main" containerID="cri-o://8efca794defe05169b3c9869df6bcc4d1760a8ff814d52d26cb96d498fec2a22" gracePeriod=45
Jul 28 02:57:23.105807 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.105779    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:57:23.131622 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.131593    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nfkv\" (UniqueName: \"kubernetes.io/projected/1965ea60-0e79-4e24-b214-9b4048eb6dbe-kube-api-access-7nfkv\") pod \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") "
Jul 28 02:57:23.131799 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.131664    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-model-cache\") pod \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") "
Jul 28 02:57:23.131799 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.131712    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-kserve-provision-location\") pod \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") "
Jul 28 02:57:23.131799 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.131751    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-dshm\") pod \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") "
Jul 28 02:57:23.131799 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.131782    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-tmp-dir\") pod \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") "
Jul 28 02:57:23.132024 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.131823    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1965ea60-0e79-4e24-b214-9b4048eb6dbe-tls-certs\") pod \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") "
Jul 28 02:57:23.132174 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.132094    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-model-cache" (OuterVolumeSpecName: "model-cache") pod "1965ea60-0e79-4e24-b214-9b4048eb6dbe" (UID: "1965ea60-0e79-4e24-b214-9b4048eb6dbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:57:23.134312 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.134276    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1965ea60-0e79-4e24-b214-9b4048eb6dbe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1965ea60-0e79-4e24-b214-9b4048eb6dbe" (UID: "1965ea60-0e79-4e24-b214-9b4048eb6dbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:57:23.134451 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.134405    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-dshm" (OuterVolumeSpecName: "dshm") pod "1965ea60-0e79-4e24-b214-9b4048eb6dbe" (UID: "1965ea60-0e79-4e24-b214-9b4048eb6dbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:57:23.134509 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.134474    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1965ea60-0e79-4e24-b214-9b4048eb6dbe-kube-api-access-7nfkv" (OuterVolumeSpecName: "kube-api-access-7nfkv") pod "1965ea60-0e79-4e24-b214-9b4048eb6dbe" (UID: "1965ea60-0e79-4e24-b214-9b4048eb6dbe"). InnerVolumeSpecName "kube-api-access-7nfkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:57:23.144803 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.144749    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1965ea60-0e79-4e24-b214-9b4048eb6dbe" (UID: "1965ea60-0e79-4e24-b214-9b4048eb6dbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:57:23.203422 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.203383    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1965ea60-0e79-4e24-b214-9b4048eb6dbe" (UID: "1965ea60-0e79-4e24-b214-9b4048eb6dbe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:57:23.232461 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.232433    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-home\") pod \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\" (UID: \"1965ea60-0e79-4e24-b214-9b4048eb6dbe\") "
Jul 28 02:57:23.232649 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.232605    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-kserve-provision-location\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:57:23.232649 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.232619    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:57:23.232649 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.232629    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:57:23.232649 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.232636    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1965ea60-0e79-4e24-b214-9b4048eb6dbe-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:57:23.232649 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.232646    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nfkv\" (UniqueName: \"kubernetes.io/projected/1965ea60-0e79-4e24-b214-9b4048eb6dbe-kube-api-access-7nfkv\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:57:23.232846 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.232654    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:57:23.232984 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.232948    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-home" (OuterVolumeSpecName: "home") pod "1965ea60-0e79-4e24-b214-9b4048eb6dbe" (UID: "1965ea60-0e79-4e24-b214-9b4048eb6dbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:57:23.333042 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.333015    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1965ea60-0e79-4e24-b214-9b4048eb6dbe-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 02:57:23.781671 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.781636    2578 generic.go:356] "Generic (PLEG): container finished" podID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" containerID="8efca794defe05169b3c9869df6bcc4d1760a8ff814d52d26cb96d498fec2a22" exitCode=0
Jul 28 02:57:23.782188 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.781696    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"
Jul 28 02:57:23.782188 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.781717    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw" event={"ID":"1965ea60-0e79-4e24-b214-9b4048eb6dbe","Type":"ContainerDied","Data":"8efca794defe05169b3c9869df6bcc4d1760a8ff814d52d26cb96d498fec2a22"}
Jul 28 02:57:23.782188 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.781791    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw" event={"ID":"1965ea60-0e79-4e24-b214-9b4048eb6dbe","Type":"ContainerDied","Data":"a436869f90a0bea523ab602b691789848acb5e3712653a5e221f817084253c6e"}
Jul 28 02:57:23.782188 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.781831    2578 scope.go:117] "RemoveContainer" containerID="8efca794defe05169b3c9869df6bcc4d1760a8ff814d52d26cb96d498fec2a22"
Jul 28 02:57:23.790572 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.790554    2578 scope.go:117] "RemoveContainer" containerID="153a940b79f091cb46c457654ca40a06e70948195bcd7fbf02d3edb2f4957b5c"
Jul 28 02:57:23.798093 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.798075    2578 scope.go:117] "RemoveContainer" containerID="8efca794defe05169b3c9869df6bcc4d1760a8ff814d52d26cb96d498fec2a22"
Jul 28 02:57:23.798372 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:57:23.798353    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8efca794defe05169b3c9869df6bcc4d1760a8ff814d52d26cb96d498fec2a22\": container with ID starting with 8efca794defe05169b3c9869df6bcc4d1760a8ff814d52d26cb96d498fec2a22 not found: ID does not exist" containerID="8efca794defe05169b3c9869df6bcc4d1760a8ff814d52d26cb96d498fec2a22"
Jul 28 02:57:23.798422 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.798380    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8efca794defe05169b3c9869df6bcc4d1760a8ff814d52d26cb96d498fec2a22"} err="failed to get container status \"8efca794defe05169b3c9869df6bcc4d1760a8ff814d52d26cb96d498fec2a22\": rpc error: code = NotFound desc = could not find container \"8efca794defe05169b3c9869df6bcc4d1760a8ff814d52d26cb96d498fec2a22\": container with ID starting with 8efca794defe05169b3c9869df6bcc4d1760a8ff814d52d26cb96d498fec2a22 not found: ID does not exist"
Jul 28 02:57:23.798422 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.798397    2578 scope.go:117] "RemoveContainer" containerID="153a940b79f091cb46c457654ca40a06e70948195bcd7fbf02d3edb2f4957b5c"
Jul 28 02:57:23.798654 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:57:23.798636    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153a940b79f091cb46c457654ca40a06e70948195bcd7fbf02d3edb2f4957b5c\": container with ID starting with 153a940b79f091cb46c457654ca40a06e70948195bcd7fbf02d3edb2f4957b5c not found: ID does not exist" containerID="153a940b79f091cb46c457654ca40a06e70948195bcd7fbf02d3edb2f4957b5c"
Jul 28 02:57:23.798698 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.798659    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153a940b79f091cb46c457654ca40a06e70948195bcd7fbf02d3edb2f4957b5c"} err="failed to get container status \"153a940b79f091cb46c457654ca40a06e70948195bcd7fbf02d3edb2f4957b5c\": rpc error: code = NotFound desc = could not find container \"153a940b79f091cb46c457654ca40a06e70948195bcd7fbf02d3edb2f4957b5c\": container with ID starting with 153a940b79f091cb46c457654ca40a06e70948195bcd7fbf02d3edb2f4957b5c not found: ID does not exist"
Jul 28 02:57:23.802587 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.802566    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"]
Jul 28 02:57:23.805761 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:23.805742    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-b4c7b9b88-wrpzw"]
Jul 28 02:57:24.985567 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:24.985532    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" path="/var/lib/kubelet/pods/1965ea60-0e79-4e24-b214-9b4048eb6dbe/volumes"
Jul 28 02:57:58.548143 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.548110    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"]
Jul 28 02:57:58.548566 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.548416    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" containerName="main"
Jul 28 02:57:58.548566 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.548428    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" containerName="main"
Jul 28 02:57:58.548566 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.548450    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" containerName="storage-initializer"
Jul 28 02:57:58.548566 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.548455    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" containerName="storage-initializer"
Jul 28 02:57:58.548566 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.548506    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1965ea60-0e79-4e24-b214-9b4048eb6dbe" containerName="main"
Jul 28 02:57:58.552546 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.552530    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.555258 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.555238    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 28 02:57:58.555397 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.555239    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Jul 28 02:57:58.555397 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.555386    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 28 02:57:58.555531 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.555411    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 28 02:57:58.555531 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.555517    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-psdvf\""
Jul 28 02:57:58.562374 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.562353    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"]
Jul 28 02:57:58.584819 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.584787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-kserve-provision-location\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.584938 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.584831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-model-cache\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.584938 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.584899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce91b9e7-ec8b-4782-aac4-881b774b517b-tls-certs\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.584938 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.584928    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwrcn\" (UniqueName: \"kubernetes.io/projected/ce91b9e7-ec8b-4782-aac4-881b774b517b-kube-api-access-pwrcn\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.585065 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.584966    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-dshm\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.585065 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.584999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-home\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.585065 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.585025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-tmp-dir\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.685622 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.685600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-tmp-dir\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.685749 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.685633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-kserve-provision-location\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.685749 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.685657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-model-cache\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.685749 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.685696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce91b9e7-ec8b-4782-aac4-881b774b517b-tls-certs\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.685749 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.685712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwrcn\" (UniqueName: \"kubernetes.io/projected/ce91b9e7-ec8b-4782-aac4-881b774b517b-kube-api-access-pwrcn\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.685749 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.685741    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-dshm\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.686009 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.685773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-home\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.686009 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.685990    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-tmp-dir\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.686112 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.686040    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-kserve-provision-location\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.686112 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.686103    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-model-cache\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.686226 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.686195    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-home\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.687942 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.687926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-dshm\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.688164 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.688146    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce91b9e7-ec8b-4782-aac4-881b774b517b-tls-certs\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.693994 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.693976    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwrcn\" (UniqueName: \"kubernetes.io/projected/ce91b9e7-ec8b-4782-aac4-881b774b517b-kube-api-access-pwrcn\") pod \"prestop-hook-test-kserve-54c766cd44-s48xz\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.863396 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.863299    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:57:58.998765 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:58.998737    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"]
Jul 28 02:57:59.002528 ip-10-0-139-81 kubenswrapper[2578]: W0728 02:57:59.002505    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce91b9e7_ec8b_4782_aac4_881b774b517b.slice/crio-4bb0c23283966ffcfda60fc176db5fb99623151d9090ed24f5b8856eb026acb1 WatchSource:0}: Error finding container 4bb0c23283966ffcfda60fc176db5fb99623151d9090ed24f5b8856eb026acb1: Status 404 returned error can't find the container with id 4bb0c23283966ffcfda60fc176db5fb99623151d9090ed24f5b8856eb026acb1
Jul 28 02:57:59.945188 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:59.945147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" event={"ID":"ce91b9e7-ec8b-4782-aac4-881b774b517b","Type":"ContainerStarted","Data":"78dba2e0f5d21e12c7ba874de65cb698a1008f8783657663f813244223ea7e0f"}
Jul 28 02:57:59.945188 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:57:59.945197    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" event={"ID":"ce91b9e7-ec8b-4782-aac4-881b774b517b","Type":"ContainerStarted","Data":"4bb0c23283966ffcfda60fc176db5fb99623151d9090ed24f5b8856eb026acb1"}
Jul 28 02:58:03.966341 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:58:03.966294    2578 generic.go:356] "Generic (PLEG): container finished" podID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerID="78dba2e0f5d21e12c7ba874de65cb698a1008f8783657663f813244223ea7e0f" exitCode=0
Jul 28 02:58:03.966769 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:58:03.966348    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" event={"ID":"ce91b9e7-ec8b-4782-aac4-881b774b517b","Type":"ContainerDied","Data":"78dba2e0f5d21e12c7ba874de65cb698a1008f8783657663f813244223ea7e0f"}
Jul 28 02:58:04.973942 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:58:04.973901    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" event={"ID":"ce91b9e7-ec8b-4782-aac4-881b774b517b","Type":"ContainerStarted","Data":"41c388db8728152d90d4d7ca0792e0c58424be36fa31a7fd48a717842f2c38d2"}
Jul 28 02:58:04.996512 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:58:04.996454    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" podStartSLOduration=2.233000202 podStartE2EDuration="6.996434171s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.763433969s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:58:04.994230315 +0000 UTC m=+5528.571687945" watchObservedRunningTime="2026-07-28 02:58:04.996434171 +0000 UTC m=+5528.573891792"
Jul 28 02:58:05.863937 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:58:05.863898    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:58:08.864131 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:58:08.864088    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:58:08.865956 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:58:08.865931    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" podUID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 02:58:18.863957 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:58:18.863899    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" podUID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 02:58:20.486256 ip-10-0-139-81 kubenswrapper[2578]: E0728 02:58:20.486212    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 02:58:21.053799 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:58:21.053769    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 02:58:28.864247 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:58:28.864199    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" podUID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 02:58:38.863975 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:58:38.863926    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" podUID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 02:58:48.864354 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:58:48.864233    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" podUID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 02:58:58.864728 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:58:58.864684    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" podUID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 02:59:08.864396 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:08.864307    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" podUID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 02:59:18.863731 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:18.863685    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" podUID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 02:59:28.864195 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:28.864142    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" podUID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 02:59:38.864738 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:38.864686    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" podUID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 02:59:48.864530 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:48.864490    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" podUID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 02:59:58.873720 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:58.873686    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:59:58.881974 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:58.881943    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 02:59:59.390579 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.390545    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"]
Jul 28 02:59:59.416349 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.416302    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"]
Jul 28 02:59:59.419566 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.419550    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.431661 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.431637    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"]
Jul 28 02:59:59.443130 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.443100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-kserve-provision-location\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.443251 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.443137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b36af28c-c369-4572-867b-76ccb22adf5e-tls-certs\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.443251 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.443173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-home\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.443251 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.443194    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-tmp-dir\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.443392 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.443277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-dshm\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.443392 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.443302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmj9c\" (UniqueName: \"kubernetes.io/projected/b36af28c-c369-4572-867b-76ccb22adf5e-kube-api-access-nmj9c\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.443392 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.443373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-model-cache\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.543880 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.543843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-dshm\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.543880 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.543882    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmj9c\" (UniqueName: \"kubernetes.io/projected/b36af28c-c369-4572-867b-76ccb22adf5e-kube-api-access-nmj9c\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.544128 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.543927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-model-cache\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.544128 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.543950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-kserve-provision-location\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.544128 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.543966    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b36af28c-c369-4572-867b-76ccb22adf5e-tls-certs\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.544128 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.544014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-home\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.544128 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.544038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-tmp-dir\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.544418 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.544377    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-model-cache\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.544479 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.544423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-tmp-dir\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.544562 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.544536    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-kserve-provision-location\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.544620 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.544557    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-home\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.546550 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.546528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-dshm\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.546666 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.546642    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b36af28c-c369-4572-867b-76ccb22adf5e-tls-certs\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.552052 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.552027    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmj9c\" (UniqueName: \"kubernetes.io/projected/b36af28c-c369-4572-867b-76ccb22adf5e-kube-api-access-nmj9c\") pod \"prestop-hook-test-kserve-54c766cd44-fhkjx\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.729692 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.729611    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 02:59:59.862918 ip-10-0-139-81 kubenswrapper[2578]: I0728 02:59:59.862896    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"]
Jul 28 02:59:59.864268 ip-10-0-139-81 kubenswrapper[2578]: W0728 02:59:59.864242    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36af28c_c369_4572_867b_76ccb22adf5e.slice/crio-c962a6a0ceca5f9b695039c79ab748cbd130a590585f1730fc805288358c39eb WatchSource:0}: Error finding container c962a6a0ceca5f9b695039c79ab748cbd130a590585f1730fc805288358c39eb: Status 404 returned error can't find the container with id c962a6a0ceca5f9b695039c79ab748cbd130a590585f1730fc805288358c39eb
Jul 28 03:00:00.511772 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:00.511735    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx" event={"ID":"b36af28c-c369-4572-867b-76ccb22adf5e","Type":"ContainerStarted","Data":"f91233f16f72880260d4e55c2456af61e7f5f9c4aa2e59afae073e39523659d2"}
Jul 28 03:00:00.512240 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:00.511795    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx" event={"ID":"b36af28c-c369-4572-867b-76ccb22adf5e","Type":"ContainerStarted","Data":"c962a6a0ceca5f9b695039c79ab748cbd130a590585f1730fc805288358c39eb"}
Jul 28 03:00:04.537822 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:04.537733    2578 generic.go:356] "Generic (PLEG): container finished" podID="b36af28c-c369-4572-867b-76ccb22adf5e" containerID="f91233f16f72880260d4e55c2456af61e7f5f9c4aa2e59afae073e39523659d2" exitCode=0
Jul 28 03:00:04.537822 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:04.537807    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx" event={"ID":"b36af28c-c369-4572-867b-76ccb22adf5e","Type":"ContainerDied","Data":"f91233f16f72880260d4e55c2456af61e7f5f9c4aa2e59afae073e39523659d2"}
Jul 28 03:00:05.545111 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:05.545079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx" event={"ID":"b36af28c-c369-4572-867b-76ccb22adf5e","Type":"ContainerStarted","Data":"1449e83becadf7e387db220d6d3a1254b2ed3766f77ffe0e5d4ef809f52f9a7c"}
Jul 28 03:00:05.567835 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:05.567785    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx" podStartSLOduration=2.211268761 podStartE2EDuration="6.567769266s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.356500505s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 03:00:05.566419161 +0000 UTC m=+5649.143876779" watchObservedRunningTime="2026-07-28 03:00:05.567769266 +0000 UTC m=+5649.145226902"
Jul 28 03:00:05.730521 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:05.730487    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 03:00:09.730601 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:09.730110    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 03:00:09.731712 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:09.731682    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx" podUID="b36af28c-c369-4572-867b-76ccb22adf5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 28 03:00:15.534250 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:15.534211    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" podUID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerName="main" containerID="cri-o://41c388db8728152d90d4d7ca0792e0c58424be36fa31a7fd48a717842f2c38d2" gracePeriod=45
Jul 28 03:00:15.874394 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:15.874277    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" podUID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 28 03:00:16.290643 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.290614    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 03:00:16.391621 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.391580    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce91b9e7-ec8b-4782-aac4-881b774b517b-tls-certs\") pod \"ce91b9e7-ec8b-4782-aac4-881b774b517b\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") "
Jul 28 03:00:16.391621 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.391626    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-home\") pod \"ce91b9e7-ec8b-4782-aac4-881b774b517b\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") "
Jul 28 03:00:16.391885 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.391675    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-model-cache\") pod \"ce91b9e7-ec8b-4782-aac4-881b774b517b\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") "
Jul 28 03:00:16.392264 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.392207    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-kserve-provision-location\") pod \"ce91b9e7-ec8b-4782-aac4-881b774b517b\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") "
Jul 28 03:00:16.392412 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.392265    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwrcn\" (UniqueName: \"kubernetes.io/projected/ce91b9e7-ec8b-4782-aac4-881b774b517b-kube-api-access-pwrcn\") pod \"ce91b9e7-ec8b-4782-aac4-881b774b517b\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") "
Jul 28 03:00:16.392412 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.392315    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-dshm\") pod \"ce91b9e7-ec8b-4782-aac4-881b774b517b\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") "
Jul 28 03:00:16.392412 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.392377    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-tmp-dir\") pod \"ce91b9e7-ec8b-4782-aac4-881b774b517b\" (UID: \"ce91b9e7-ec8b-4782-aac4-881b774b517b\") "
Jul 28 03:00:16.392851 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.392820    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce91b9e7-ec8b-4782-aac4-881b774b517b" (UID: "ce91b9e7-ec8b-4782-aac4-881b774b517b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:00:16.392953 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.392912    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-home" (OuterVolumeSpecName: "home") pod "ce91b9e7-ec8b-4782-aac4-881b774b517b" (UID: "ce91b9e7-ec8b-4782-aac4-881b774b517b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:00:16.395547 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.393393    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:00:16.395547 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.393422    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:00:16.402023 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.395902    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce91b9e7-ec8b-4782-aac4-881b774b517b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce91b9e7-ec8b-4782-aac4-881b774b517b" (UID: "ce91b9e7-ec8b-4782-aac4-881b774b517b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 03:00:16.403080 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.402733    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-dshm" (OuterVolumeSpecName: "dshm") pod "ce91b9e7-ec8b-4782-aac4-881b774b517b" (UID: "ce91b9e7-ec8b-4782-aac4-881b774b517b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:00:16.403202 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.403078    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce91b9e7-ec8b-4782-aac4-881b774b517b-kube-api-access-pwrcn" (OuterVolumeSpecName: "kube-api-access-pwrcn") pod "ce91b9e7-ec8b-4782-aac4-881b774b517b" (UID: "ce91b9e7-ec8b-4782-aac4-881b774b517b"). InnerVolumeSpecName "kube-api-access-pwrcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 03:00:16.406659 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.406638    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce91b9e7-ec8b-4782-aac4-881b774b517b" (UID: "ce91b9e7-ec8b-4782-aac4-881b774b517b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:00:16.480094 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.480043    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce91b9e7-ec8b-4782-aac4-881b774b517b" (UID: "ce91b9e7-ec8b-4782-aac4-881b774b517b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:00:16.494749 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.494711    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:00:16.494749 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.494741    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce91b9e7-ec8b-4782-aac4-881b774b517b-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:00:16.494749 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.494756    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-kserve-provision-location\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:00:16.495032 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.494766    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwrcn\" (UniqueName: \"kubernetes.io/projected/ce91b9e7-ec8b-4782-aac4-881b774b517b-kube-api-access-pwrcn\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:00:16.495032 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.494775    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce91b9e7-ec8b-4782-aac4-881b774b517b-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:00:16.602778 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.602741    2578 generic.go:356] "Generic (PLEG): container finished" podID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerID="41c388db8728152d90d4d7ca0792e0c58424be36fa31a7fd48a717842f2c38d2" exitCode=0
Jul 28 03:00:16.603215 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.602825    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"
Jul 28 03:00:16.603215 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.602819    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" event={"ID":"ce91b9e7-ec8b-4782-aac4-881b774b517b","Type":"ContainerDied","Data":"41c388db8728152d90d4d7ca0792e0c58424be36fa31a7fd48a717842f2c38d2"}
Jul 28 03:00:16.603215 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.603001    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz" event={"ID":"ce91b9e7-ec8b-4782-aac4-881b774b517b","Type":"ContainerDied","Data":"4bb0c23283966ffcfda60fc176db5fb99623151d9090ed24f5b8856eb026acb1"}
Jul 28 03:00:16.603215 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.603047    2578 scope.go:117] "RemoveContainer" containerID="41c388db8728152d90d4d7ca0792e0c58424be36fa31a7fd48a717842f2c38d2"
Jul 28 03:00:16.613422 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.613314    2578 scope.go:117] "RemoveContainer" containerID="78dba2e0f5d21e12c7ba874de65cb698a1008f8783657663f813244223ea7e0f"
Jul 28 03:00:16.624740 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.624722    2578 scope.go:117] "RemoveContainer" containerID="41c388db8728152d90d4d7ca0792e0c58424be36fa31a7fd48a717842f2c38d2"
Jul 28 03:00:16.625433 ip-10-0-139-81 kubenswrapper[2578]: E0728 03:00:16.625302    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c388db8728152d90d4d7ca0792e0c58424be36fa31a7fd48a717842f2c38d2\": container with ID starting with 41c388db8728152d90d4d7ca0792e0c58424be36fa31a7fd48a717842f2c38d2 not found: ID does not exist" containerID="41c388db8728152d90d4d7ca0792e0c58424be36fa31a7fd48a717842f2c38d2"
Jul 28 03:00:16.625433 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.625352    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c388db8728152d90d4d7ca0792e0c58424be36fa31a7fd48a717842f2c38d2"} err="failed to get container status \"41c388db8728152d90d4d7ca0792e0c58424be36fa31a7fd48a717842f2c38d2\": rpc error: code = NotFound desc = could not find container \"41c388db8728152d90d4d7ca0792e0c58424be36fa31a7fd48a717842f2c38d2\": container with ID starting with 41c388db8728152d90d4d7ca0792e0c58424be36fa31a7fd48a717842f2c38d2 not found: ID does not exist"
Jul 28 03:00:16.625433 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.625377    2578 scope.go:117] "RemoveContainer" containerID="78dba2e0f5d21e12c7ba874de65cb698a1008f8783657663f813244223ea7e0f"
Jul 28 03:00:16.625690 ip-10-0-139-81 kubenswrapper[2578]: E0728 03:00:16.625640    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78dba2e0f5d21e12c7ba874de65cb698a1008f8783657663f813244223ea7e0f\": container with ID starting with 78dba2e0f5d21e12c7ba874de65cb698a1008f8783657663f813244223ea7e0f not found: ID does not exist" containerID="78dba2e0f5d21e12c7ba874de65cb698a1008f8783657663f813244223ea7e0f"
Jul 28 03:00:16.625690 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.625677    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78dba2e0f5d21e12c7ba874de65cb698a1008f8783657663f813244223ea7e0f"} err="failed to get container status \"78dba2e0f5d21e12c7ba874de65cb698a1008f8783657663f813244223ea7e0f\": rpc error: code = NotFound desc = could not find container \"78dba2e0f5d21e12c7ba874de65cb698a1008f8783657663f813244223ea7e0f\": container with ID starting with 78dba2e0f5d21e12c7ba874de65cb698a1008f8783657663f813244223ea7e0f not found: ID does not exist"
Jul 28 03:00:16.640765 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.640745    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"]
Jul 28 03:00:16.645878 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.645849    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-s48xz"]
Jul 28 03:00:16.987195 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:16.987158    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce91b9e7-ec8b-4782-aac4-881b774b517b" path="/var/lib/kubelet/pods/ce91b9e7-ec8b-4782-aac4-881b774b517b/volumes"
Jul 28 03:00:19.730070 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:19.729973    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx" podUID="b36af28c-c369-4572-867b-76ccb22adf5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 28 03:00:24.054737 ip-10-0-139-81 kubenswrapper[2578]: E0728 03:00:24.054683    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 03:00:24.643996 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:24.643965    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 03:00:29.730222 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:29.730181    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx" podUID="b36af28c-c369-4572-867b-76ccb22adf5e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 28 03:00:35.422758 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:35.422717    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"]
Jul 28 03:00:50.445279 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:50.445236    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx" podUID="b36af28c-c369-4572-867b-76ccb22adf5e" containerName="main" containerID="cri-o://1449e83becadf7e387db220d6d3a1254b2ed3766f77ffe0e5d4ef809f52f9a7c" gracePeriod=45
Jul 28 03:00:52.286596 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.286573    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-54c766cd44-fhkjx_b36af28c-c369-4572-867b-76ccb22adf5e/main/0.log"
Jul 28 03:00:52.288325 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.288303    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 03:00:52.321627 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.321602    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-home\") pod \"b36af28c-c369-4572-867b-76ccb22adf5e\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") "
Jul 28 03:00:52.321764 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.321654    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-tmp-dir\") pod \"b36af28c-c369-4572-867b-76ccb22adf5e\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") "
Jul 28 03:00:52.321764 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.321684    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmj9c\" (UniqueName: \"kubernetes.io/projected/b36af28c-c369-4572-867b-76ccb22adf5e-kube-api-access-nmj9c\") pod \"b36af28c-c369-4572-867b-76ccb22adf5e\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") "
Jul 28 03:00:52.321764 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.321707    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b36af28c-c369-4572-867b-76ccb22adf5e-tls-certs\") pod \"b36af28c-c369-4572-867b-76ccb22adf5e\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") "
Jul 28 03:00:52.321764 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.321738    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-model-cache\") pod \"b36af28c-c369-4572-867b-76ccb22adf5e\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") "
Jul 28 03:00:52.321922 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.321780    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-dshm\") pod \"b36af28c-c369-4572-867b-76ccb22adf5e\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") "
Jul 28 03:00:52.321922 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.321811    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-kserve-provision-location\") pod \"b36af28c-c369-4572-867b-76ccb22adf5e\" (UID: \"b36af28c-c369-4572-867b-76ccb22adf5e\") "
Jul 28 03:00:52.321922 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.321891    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-home" (OuterVolumeSpecName: "home") pod "b36af28c-c369-4572-867b-76ccb22adf5e" (UID: "b36af28c-c369-4572-867b-76ccb22adf5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:00:52.322354 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.322083    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:00:52.322354 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.322130    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "b36af28c-c369-4572-867b-76ccb22adf5e" (UID: "b36af28c-c369-4572-867b-76ccb22adf5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:00:52.322354 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.322253    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b36af28c-c369-4572-867b-76ccb22adf5e" (UID: "b36af28c-c369-4572-867b-76ccb22adf5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:00:52.324111 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.324084    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-dshm" (OuterVolumeSpecName: "dshm") pod "b36af28c-c369-4572-867b-76ccb22adf5e" (UID: "b36af28c-c369-4572-867b-76ccb22adf5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:00:52.324399 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.324376    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36af28c-c369-4572-867b-76ccb22adf5e-kube-api-access-nmj9c" (OuterVolumeSpecName: "kube-api-access-nmj9c") pod "b36af28c-c369-4572-867b-76ccb22adf5e" (UID: "b36af28c-c369-4572-867b-76ccb22adf5e"). InnerVolumeSpecName "kube-api-access-nmj9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 03:00:52.324486 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.324423    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b36af28c-c369-4572-867b-76ccb22adf5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b36af28c-c369-4572-867b-76ccb22adf5e" (UID: "b36af28c-c369-4572-867b-76ccb22adf5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 03:00:52.379982 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.379909    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b36af28c-c369-4572-867b-76ccb22adf5e" (UID: "b36af28c-c369-4572-867b-76ccb22adf5e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:00:52.422685 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.422658    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:00:52.422685 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.422683    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmj9c\" (UniqueName: \"kubernetes.io/projected/b36af28c-c369-4572-867b-76ccb22adf5e-kube-api-access-nmj9c\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:00:52.422685 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.422692    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b36af28c-c369-4572-867b-76ccb22adf5e-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:00:52.422948 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.422701    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:00:52.422948 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.422708    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:00:52.422948 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.422717    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b36af28c-c369-4572-867b-76ccb22adf5e-kserve-provision-location\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:00:52.772462 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.772437    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-54c766cd44-fhkjx_b36af28c-c369-4572-867b-76ccb22adf5e/main/0.log"
Jul 28 03:00:52.772854 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.772830    2578 generic.go:356] "Generic (PLEG): container finished" podID="b36af28c-c369-4572-867b-76ccb22adf5e" containerID="1449e83becadf7e387db220d6d3a1254b2ed3766f77ffe0e5d4ef809f52f9a7c" exitCode=130
Jul 28 03:00:52.772931 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.772895    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"
Jul 28 03:00:52.772931 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.772913    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx" event={"ID":"b36af28c-c369-4572-867b-76ccb22adf5e","Type":"ContainerDied","Data":"1449e83becadf7e387db220d6d3a1254b2ed3766f77ffe0e5d4ef809f52f9a7c"}
Jul 28 03:00:52.773038 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.772966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx" event={"ID":"b36af28c-c369-4572-867b-76ccb22adf5e","Type":"ContainerDied","Data":"c962a6a0ceca5f9b695039c79ab748cbd130a590585f1730fc805288358c39eb"}
Jul 28 03:00:52.773038 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.772989    2578 scope.go:117] "RemoveContainer" containerID="1449e83becadf7e387db220d6d3a1254b2ed3766f77ffe0e5d4ef809f52f9a7c"
Jul 28 03:00:52.783520 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.783504    2578 scope.go:117] "RemoveContainer" containerID="f91233f16f72880260d4e55c2456af61e7f5f9c4aa2e59afae073e39523659d2"
Jul 28 03:00:52.790939 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.790922    2578 scope.go:117] "RemoveContainer" containerID="1449e83becadf7e387db220d6d3a1254b2ed3766f77ffe0e5d4ef809f52f9a7c"
Jul 28 03:00:52.791208 ip-10-0-139-81 kubenswrapper[2578]: E0728 03:00:52.791189    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1449e83becadf7e387db220d6d3a1254b2ed3766f77ffe0e5d4ef809f52f9a7c\": container with ID starting with 1449e83becadf7e387db220d6d3a1254b2ed3766f77ffe0e5d4ef809f52f9a7c not found: ID does not exist" containerID="1449e83becadf7e387db220d6d3a1254b2ed3766f77ffe0e5d4ef809f52f9a7c"
Jul 28 03:00:52.791279 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.791220    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1449e83becadf7e387db220d6d3a1254b2ed3766f77ffe0e5d4ef809f52f9a7c"} err="failed to get container status \"1449e83becadf7e387db220d6d3a1254b2ed3766f77ffe0e5d4ef809f52f9a7c\": rpc error: code = NotFound desc = could not find container \"1449e83becadf7e387db220d6d3a1254b2ed3766f77ffe0e5d4ef809f52f9a7c\": container with ID starting with 1449e83becadf7e387db220d6d3a1254b2ed3766f77ffe0e5d4ef809f52f9a7c not found: ID does not exist"
Jul 28 03:00:52.791279 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.791244    2578 scope.go:117] "RemoveContainer" containerID="f91233f16f72880260d4e55c2456af61e7f5f9c4aa2e59afae073e39523659d2"
Jul 28 03:00:52.791471 ip-10-0-139-81 kubenswrapper[2578]: E0728 03:00:52.791455    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91233f16f72880260d4e55c2456af61e7f5f9c4aa2e59afae073e39523659d2\": container with ID starting with f91233f16f72880260d4e55c2456af61e7f5f9c4aa2e59afae073e39523659d2 not found: ID does not exist" containerID="f91233f16f72880260d4e55c2456af61e7f5f9c4aa2e59afae073e39523659d2"
Jul 28 03:00:52.791527 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.791479    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91233f16f72880260d4e55c2456af61e7f5f9c4aa2e59afae073e39523659d2"} err="failed to get container status \"f91233f16f72880260d4e55c2456af61e7f5f9c4aa2e59afae073e39523659d2\": rpc error: code = NotFound desc = could not find container \"f91233f16f72880260d4e55c2456af61e7f5f9c4aa2e59afae073e39523659d2\": container with ID starting with f91233f16f72880260d4e55c2456af61e7f5f9c4aa2e59afae073e39523659d2 not found: ID does not exist"
Jul 28 03:00:52.796266 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.796245    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"]
Jul 28 03:00:52.799674 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.799651    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-fhkjx"]
Jul 28 03:00:52.986275 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:00:52.986244    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b36af28c-c369-4572-867b-76ccb22adf5e" path="/var/lib/kubelet/pods/b36af28c-c369-4572-867b-76ccb22adf5e/volumes"
Jul 28 03:01:12.723498 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.723459    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"]
Jul 28 03:01:12.724021 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.723781    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerName="storage-initializer"
Jul 28 03:01:12.724021 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.723793    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerName="storage-initializer"
Jul 28 03:01:12.724021 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.723801    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerName="main"
Jul 28 03:01:12.724021 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.723807    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerName="main"
Jul 28 03:01:12.724021 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.723824    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b36af28c-c369-4572-867b-76ccb22adf5e" containerName="storage-initializer"
Jul 28 03:01:12.724021 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.723831    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b36af28c-c369-4572-867b-76ccb22adf5e" containerName="storage-initializer"
Jul 28 03:01:12.724021 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.723841    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b36af28c-c369-4572-867b-76ccb22adf5e" containerName="main"
Jul 28 03:01:12.724021 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.723845    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b36af28c-c369-4572-867b-76ccb22adf5e" containerName="main"
Jul 28 03:01:12.724021 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.723898    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce91b9e7-ec8b-4782-aac4-881b774b517b" containerName="main"
Jul 28 03:01:12.724021 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.723905    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b36af28c-c369-4572-867b-76ccb22adf5e" containerName="main"
Jul 28 03:01:12.726815 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.726793    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:12.731126 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.731106    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-7q545\""
Jul 28 03:01:12.731126 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.731114    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Jul 28 03:01:12.731304 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.731105    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 28 03:01:12.731304 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.731158    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Jul 28 03:01:12.736573 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.736545    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"]
Jul 28 03:01:12.793702 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.793670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-ssg2k\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:12.793839 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.793720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-ssg2k\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:12.793839 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.793746    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-ssg2k\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:12.793839 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.793786    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47abb7ef-b2ac-47d2-ac98-0820240c57f8-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-ssg2k\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:12.793839 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.793801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-ssg2k\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:12.793987 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.793876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpb78\" (UniqueName: \"kubernetes.io/projected/47abb7ef-b2ac-47d2-ac98-0820240c57f8-kube-api-access-mpb78\") pod \"rolling-upgrade-test-kserve-74b79d7f77-ssg2k\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:12.894597 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.894567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-ssg2k\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:12.894769 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.894606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-ssg2k\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:12.894769 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.894626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-ssg2k\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:12.894769 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.894656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47abb7ef-b2ac-47d2-ac98-0820240c57f8-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-ssg2k\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:12.894931 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.894767    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-ssg2k\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:12.894931 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.894808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpb78\" (UniqueName: \"kubernetes.io/projected/47abb7ef-b2ac-47d2-ac98-0820240c57f8-kube-api-access-mpb78\") pod \"rolling-upgrade-test-kserve-74b79d7f77-ssg2k\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:12.895048 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.895030    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-ssg2k\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:12.895106 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.895070    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-ssg2k\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:12.895147 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.895134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-ssg2k\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:12.896992 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.896964    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-ssg2k\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:12.897168 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.897151    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47abb7ef-b2ac-47d2-ac98-0820240c57f8-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-ssg2k\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:12.902877 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:12.902850    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpb78\" (UniqueName: \"kubernetes.io/projected/47abb7ef-b2ac-47d2-ac98-0820240c57f8-kube-api-access-mpb78\") pod \"rolling-upgrade-test-kserve-74b79d7f77-ssg2k\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:13.038069 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:13.038038    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:13.164589 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:13.164553    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"]
Jul 28 03:01:13.167743 ip-10-0-139-81 kubenswrapper[2578]: W0728 03:01:13.167715    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47abb7ef_b2ac_47d2_ac98_0820240c57f8.slice/crio-599b0fbdf0a8fb05c811b3e9ad36a320fa350104069aacec8ea02befa77d085c WatchSource:0}: Error finding container 599b0fbdf0a8fb05c811b3e9ad36a320fa350104069aacec8ea02befa77d085c: Status 404 returned error can't find the container with id 599b0fbdf0a8fb05c811b3e9ad36a320fa350104069aacec8ea02befa77d085c
Jul 28 03:01:13.863269 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:13.863230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k" event={"ID":"47abb7ef-b2ac-47d2-ac98-0820240c57f8","Type":"ContainerStarted","Data":"f83af6750e5aa9363c3349c915ae7f1976f2b2308b96278469515c2aaa23df3d"}
Jul 28 03:01:13.863698 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:13.863283    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k" event={"ID":"47abb7ef-b2ac-47d2-ac98-0820240c57f8","Type":"ContainerStarted","Data":"599b0fbdf0a8fb05c811b3e9ad36a320fa350104069aacec8ea02befa77d085c"}
Jul 28 03:01:13.883553 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:13.883507    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k" podStartSLOduration=1.883493107 podStartE2EDuration="1.883493107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 03:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 03:01:13.881666769 +0000 UTC m=+5717.459124386" watchObservedRunningTime="2026-07-28 03:01:13.883493107 +0000 UTC m=+5717.460950723"
Jul 28 03:01:14.038813 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:14.038784    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:23.038796 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:23.038761    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:23.051537 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:23.051511    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:01:23.927060 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:01:23.927031    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:02:09.626786 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.626680    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"]
Jul 28 03:02:09.629895 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.629871    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:09.639859 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.639777    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"]
Jul 28 03:02:09.691781 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.691742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t7lpr\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:09.691981 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.691797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t7lpr\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:09.691981 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.691880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zg54\" (UniqueName: \"kubernetes.io/projected/71b7272c-7b24-4158-8418-f596c7b04364-kube-api-access-4zg54\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t7lpr\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:09.691981 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.691925    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t7lpr\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:09.692152 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.691999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t7lpr\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:09.692152 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.692065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b7272c-7b24-4158-8418-f596c7b04364-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t7lpr\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:09.792618 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.792580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t7lpr\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:09.792618 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.792619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t7lpr\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:09.792870 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.792644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b7272c-7b24-4158-8418-f596c7b04364-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t7lpr\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:09.792870 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.792681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t7lpr\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:09.792870 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.792711    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t7lpr\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:09.792870 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.792761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zg54\" (UniqueName: \"kubernetes.io/projected/71b7272c-7b24-4158-8418-f596c7b04364-kube-api-access-4zg54\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t7lpr\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:09.793090 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.793056    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t7lpr\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:09.793090 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.793080    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t7lpr\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:09.793180 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.793129    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t7lpr\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:09.795037 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.795008    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t7lpr\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:09.795274 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.795259    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b7272c-7b24-4158-8418-f596c7b04364-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t7lpr\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:09.800363 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.800325    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zg54\" (UniqueName: \"kubernetes.io/projected/71b7272c-7b24-4158-8418-f596c7b04364-kube-api-access-4zg54\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t7lpr\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:09.943186 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:09.943089    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:10.071435 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:10.071402    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"]
Jul 28 03:02:10.074503 ip-10-0-139-81 kubenswrapper[2578]: W0728 03:02:10.074475    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b7272c_7b24_4158_8418_f596c7b04364.slice/crio-4260cff902d4e5441dce17a323faf316fe74a92d342ccf48246fa2ff6a8b6fea WatchSource:0}: Error finding container 4260cff902d4e5441dce17a323faf316fe74a92d342ccf48246fa2ff6a8b6fea: Status 404 returned error can't find the container with id 4260cff902d4e5441dce17a323faf316fe74a92d342ccf48246fa2ff6a8b6fea
Jul 28 03:02:10.114782 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:10.114752    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr" event={"ID":"71b7272c-7b24-4158-8418-f596c7b04364","Type":"ContainerStarted","Data":"4260cff902d4e5441dce17a323faf316fe74a92d342ccf48246fa2ff6a8b6fea"}
Jul 28 03:02:11.122649 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:11.122611    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr" event={"ID":"71b7272c-7b24-4158-8418-f596c7b04364","Type":"ContainerStarted","Data":"f217a4c5ad5fbf589065137383492b6dceb413c085d4711d87a744256be3f0f3"}
Jul 28 03:02:11.140462 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:11.140411    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr" podStartSLOduration=2.140396013 podStartE2EDuration="2.140396013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 03:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 03:02:11.138843436 +0000 UTC m=+5774.716301078" watchObservedRunningTime="2026-07-28 03:02:11.140396013 +0000 UTC m=+5774.717853634"
Jul 28 03:02:11.944146 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:11.944114    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:19.943489 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:19.943445    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:19.961155 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:19.961128    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:20.185939 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.185910    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:20.231068 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.230985    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"]
Jul 28 03:02:20.240137 ip-10-0-139-81 kubenswrapper[2578]: E0728 03:02:20.240097    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 03:02:20.240137 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 03:02:20.240137 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 03:02:20.240137 ip-10-0-139-81 kubenswrapper[2578]:  > containerID="f83af6750e5aa9363c3349c915ae7f1976f2b2308b96278469515c2aaa23df3d" cmd=["/bin/sleep","15"]
Jul 28 03:02:20.240433 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.240154    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k" message="" err=<
Jul 28 03:02:20.240433 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 03:02:20.240433 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 03:02:20.240433 ip-10-0-139-81 kubenswrapper[2578]:  >
Jul 28 03:02:20.240433 ip-10-0-139-81 kubenswrapper[2578]: E0728 03:02:20.240167    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 03:02:20.240433 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 03:02:20.240433 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 03:02:20.240433 ip-10-0-139-81 kubenswrapper[2578]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k" podUID="47abb7ef-b2ac-47d2-ac98-0820240c57f8" containerName="main" containerID="cri-o://f83af6750e5aa9363c3349c915ae7f1976f2b2308b96278469515c2aaa23df3d"
Jul 28 03:02:20.240433 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.240212    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k" podUID="47abb7ef-b2ac-47d2-ac98-0820240c57f8" containerName="main" containerID="cri-o://f83af6750e5aa9363c3349c915ae7f1976f2b2308b96278469515c2aaa23df3d" gracePeriod=60
Jul 28 03:02:20.590011 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.589987    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:02:20.681692 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.681659    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-tmp-dir\") pod \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") "
Jul 28 03:02:20.681904 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.681712    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-home\") pod \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") "
Jul 28 03:02:20.681904 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.681743    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-model-cache\") pod \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") "
Jul 28 03:02:20.681904 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.681794    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpb78\" (UniqueName: \"kubernetes.io/projected/47abb7ef-b2ac-47d2-ac98-0820240c57f8-kube-api-access-mpb78\") pod \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") "
Jul 28 03:02:20.681904 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.681839    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47abb7ef-b2ac-47d2-ac98-0820240c57f8-tls-certs\") pod \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") "
Jul 28 03:02:20.681904 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.681856    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-dshm\") pod \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\" (UID: \"47abb7ef-b2ac-47d2-ac98-0820240c57f8\") "
Jul 28 03:02:20.682164 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.681966    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "47abb7ef-b2ac-47d2-ac98-0820240c57f8" (UID: "47abb7ef-b2ac-47d2-ac98-0820240c57f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:02:20.682164 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.682106    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:02:20.682280 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.682176    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-home" (OuterVolumeSpecName: "home") pod "47abb7ef-b2ac-47d2-ac98-0820240c57f8" (UID: "47abb7ef-b2ac-47d2-ac98-0820240c57f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:02:20.682280 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.682217    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "47abb7ef-b2ac-47d2-ac98-0820240c57f8" (UID: "47abb7ef-b2ac-47d2-ac98-0820240c57f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:02:20.684296 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.684272    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47abb7ef-b2ac-47d2-ac98-0820240c57f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47abb7ef-b2ac-47d2-ac98-0820240c57f8" (UID: "47abb7ef-b2ac-47d2-ac98-0820240c57f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 03:02:20.684546 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.684520    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-dshm" (OuterVolumeSpecName: "dshm") pod "47abb7ef-b2ac-47d2-ac98-0820240c57f8" (UID: "47abb7ef-b2ac-47d2-ac98-0820240c57f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:02:20.684764 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.684747    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47abb7ef-b2ac-47d2-ac98-0820240c57f8-kube-api-access-mpb78" (OuterVolumeSpecName: "kube-api-access-mpb78") pod "47abb7ef-b2ac-47d2-ac98-0820240c57f8" (UID: "47abb7ef-b2ac-47d2-ac98-0820240c57f8"). InnerVolumeSpecName "kube-api-access-mpb78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 03:02:20.783310 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.783281    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpb78\" (UniqueName: \"kubernetes.io/projected/47abb7ef-b2ac-47d2-ac98-0820240c57f8-kube-api-access-mpb78\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:02:20.783310 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.783309    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47abb7ef-b2ac-47d2-ac98-0820240c57f8-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:02:20.783522 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.783319    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:02:20.783522 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.783338    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:02:20.783522 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:20.783347    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47abb7ef-b2ac-47d2-ac98-0820240c57f8-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:02:21.180185 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:21.180081    2578 generic.go:356] "Generic (PLEG): container finished" podID="47abb7ef-b2ac-47d2-ac98-0820240c57f8" containerID="f83af6750e5aa9363c3349c915ae7f1976f2b2308b96278469515c2aaa23df3d" exitCode=0
Jul 28 03:02:21.180185 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:21.180136    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"
Jul 28 03:02:21.180754 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:21.180175    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k" event={"ID":"47abb7ef-b2ac-47d2-ac98-0820240c57f8","Type":"ContainerDied","Data":"f83af6750e5aa9363c3349c915ae7f1976f2b2308b96278469515c2aaa23df3d"}
Jul 28 03:02:21.180754 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:21.180245    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k" event={"ID":"47abb7ef-b2ac-47d2-ac98-0820240c57f8","Type":"ContainerDied","Data":"599b0fbdf0a8fb05c811b3e9ad36a320fa350104069aacec8ea02befa77d085c"}
Jul 28 03:02:21.180754 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:21.180278    2578 scope.go:117] "RemoveContainer" containerID="f83af6750e5aa9363c3349c915ae7f1976f2b2308b96278469515c2aaa23df3d"
Jul 28 03:02:21.189084 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:21.189066    2578 scope.go:117] "RemoveContainer" containerID="f83af6750e5aa9363c3349c915ae7f1976f2b2308b96278469515c2aaa23df3d"
Jul 28 03:02:21.189357 ip-10-0-139-81 kubenswrapper[2578]: E0728 03:02:21.189323    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83af6750e5aa9363c3349c915ae7f1976f2b2308b96278469515c2aaa23df3d\": container with ID starting with f83af6750e5aa9363c3349c915ae7f1976f2b2308b96278469515c2aaa23df3d not found: ID does not exist" containerID="f83af6750e5aa9363c3349c915ae7f1976f2b2308b96278469515c2aaa23df3d"
Jul 28 03:02:21.189458 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:21.189364    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83af6750e5aa9363c3349c915ae7f1976f2b2308b96278469515c2aaa23df3d"} err="failed to get container status \"f83af6750e5aa9363c3349c915ae7f1976f2b2308b96278469515c2aaa23df3d\": rpc error: code = NotFound desc = could not find container \"f83af6750e5aa9363c3349c915ae7f1976f2b2308b96278469515c2aaa23df3d\": container with ID starting with f83af6750e5aa9363c3349c915ae7f1976f2b2308b96278469515c2aaa23df3d not found: ID does not exist"
Jul 28 03:02:21.197186 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:21.197165    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"]
Jul 28 03:02:21.200742 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:21.200723    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-ssg2k"]
Jul 28 03:02:22.985260 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:22.985218    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47abb7ef-b2ac-47d2-ac98-0820240c57f8" path="/var/lib/kubelet/pods/47abb7ef-b2ac-47d2-ac98-0820240c57f8/volumes"
Jul 28 03:02:25.341423 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.341390    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"]
Jul 28 03:02:25.350300 ip-10-0-139-81 kubenswrapper[2578]: E0728 03:02:25.350259    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 03:02:25.350300 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 03:02:25.350300 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 03:02:25.350300 ip-10-0-139-81 kubenswrapper[2578]:  > containerID="f217a4c5ad5fbf589065137383492b6dceb413c085d4711d87a744256be3f0f3" cmd=["/bin/sleep","15"]
Jul 28 03:02:25.350570 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.350316    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr" message="" err=<
Jul 28 03:02:25.350570 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 03:02:25.350570 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 03:02:25.350570 ip-10-0-139-81 kubenswrapper[2578]:  >
Jul 28 03:02:25.350570 ip-10-0-139-81 kubenswrapper[2578]: E0728 03:02:25.350355    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 03:02:25.350570 ip-10-0-139-81 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 03:02:25.350570 ip-10-0-139-81 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Jul 28 03:02:25.350570 ip-10-0-139-81 kubenswrapper[2578]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr" podUID="71b7272c-7b24-4158-8418-f596c7b04364" containerName="main" containerID="cri-o://f217a4c5ad5fbf589065137383492b6dceb413c085d4711d87a744256be3f0f3"
Jul 28 03:02:25.350570 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.350395    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr" podUID="71b7272c-7b24-4158-8418-f596c7b04364" containerName="main" containerID="cri-o://f217a4c5ad5fbf589065137383492b6dceb413c085d4711d87a744256be3f0f3" gracePeriod=60
Jul 28 03:02:25.714204 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.714179    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:25.828541 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.828509    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zg54\" (UniqueName: \"kubernetes.io/projected/71b7272c-7b24-4158-8418-f596c7b04364-kube-api-access-4zg54\") pod \"71b7272c-7b24-4158-8418-f596c7b04364\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") "
Jul 28 03:02:25.828541 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.828539    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-home\") pod \"71b7272c-7b24-4158-8418-f596c7b04364\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") "
Jul 28 03:02:25.828752 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.828567    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-tmp-dir\") pod \"71b7272c-7b24-4158-8418-f596c7b04364\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") "
Jul 28 03:02:25.828752 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.828622    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-model-cache\") pod \"71b7272c-7b24-4158-8418-f596c7b04364\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") "
Jul 28 03:02:25.828752 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.828679    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-dshm\") pod \"71b7272c-7b24-4158-8418-f596c7b04364\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") "
Jul 28 03:02:25.828752 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.828720    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b7272c-7b24-4158-8418-f596c7b04364-tls-certs\") pod \"71b7272c-7b24-4158-8418-f596c7b04364\" (UID: \"71b7272c-7b24-4158-8418-f596c7b04364\") "
Jul 28 03:02:25.828915 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.828813    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-home" (OuterVolumeSpecName: "home") pod "71b7272c-7b24-4158-8418-f596c7b04364" (UID: "71b7272c-7b24-4158-8418-f596c7b04364"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:02:25.828915 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.828885    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "71b7272c-7b24-4158-8418-f596c7b04364" (UID: "71b7272c-7b24-4158-8418-f596c7b04364"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:02:25.829000 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.828927    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-model-cache" (OuterVolumeSpecName: "model-cache") pod "71b7272c-7b24-4158-8418-f596c7b04364" (UID: "71b7272c-7b24-4158-8418-f596c7b04364"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:02:25.829045 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.829022    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-home\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:02:25.829082 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.829043    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-tmp-dir\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:02:25.829082 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.829053    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-model-cache\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:02:25.830738 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.830711    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-dshm" (OuterVolumeSpecName: "dshm") pod "71b7272c-7b24-4158-8418-f596c7b04364" (UID: "71b7272c-7b24-4158-8418-f596c7b04364"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 03:02:25.830875 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.830761    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b7272c-7b24-4158-8418-f596c7b04364-kube-api-access-4zg54" (OuterVolumeSpecName: "kube-api-access-4zg54") pod "71b7272c-7b24-4158-8418-f596c7b04364" (UID: "71b7272c-7b24-4158-8418-f596c7b04364"). InnerVolumeSpecName "kube-api-access-4zg54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 03:02:25.830928 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.830885    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b7272c-7b24-4158-8418-f596c7b04364-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71b7272c-7b24-4158-8418-f596c7b04364" (UID: "71b7272c-7b24-4158-8418-f596c7b04364"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 03:02:25.929730 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.929637    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b7272c-7b24-4158-8418-f596c7b04364-dshm\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:02:25.929730 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.929665    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b7272c-7b24-4158-8418-f596c7b04364-tls-certs\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:02:25.929730 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:25.929677    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zg54\" (UniqueName: \"kubernetes.io/projected/71b7272c-7b24-4158-8418-f596c7b04364-kube-api-access-4zg54\") on node \"ip-10-0-139-81.ec2.internal\" DevicePath \"\""
Jul 28 03:02:26.207084 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:26.206979    2578 generic.go:356] "Generic (PLEG): container finished" podID="71b7272c-7b24-4158-8418-f596c7b04364" containerID="f217a4c5ad5fbf589065137383492b6dceb413c085d4711d87a744256be3f0f3" exitCode=0
Jul 28 03:02:26.207084 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:26.207038    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"
Jul 28 03:02:26.207084 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:26.207037    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr" event={"ID":"71b7272c-7b24-4158-8418-f596c7b04364","Type":"ContainerDied","Data":"f217a4c5ad5fbf589065137383492b6dceb413c085d4711d87a744256be3f0f3"}
Jul 28 03:02:26.207377 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:26.207111    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr" event={"ID":"71b7272c-7b24-4158-8418-f596c7b04364","Type":"ContainerDied","Data":"4260cff902d4e5441dce17a323faf316fe74a92d342ccf48246fa2ff6a8b6fea"}
Jul 28 03:02:26.207377 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:26.207198    2578 scope.go:117] "RemoveContainer" containerID="f217a4c5ad5fbf589065137383492b6dceb413c085d4711d87a744256be3f0f3"
Jul 28 03:02:26.217512 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:26.217486    2578 scope.go:117] "RemoveContainer" containerID="f217a4c5ad5fbf589065137383492b6dceb413c085d4711d87a744256be3f0f3"
Jul 28 03:02:26.217863 ip-10-0-139-81 kubenswrapper[2578]: E0728 03:02:26.217836    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f217a4c5ad5fbf589065137383492b6dceb413c085d4711d87a744256be3f0f3\": container with ID starting with f217a4c5ad5fbf589065137383492b6dceb413c085d4711d87a744256be3f0f3 not found: ID does not exist" containerID="f217a4c5ad5fbf589065137383492b6dceb413c085d4711d87a744256be3f0f3"
Jul 28 03:02:26.218015 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:26.217872    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f217a4c5ad5fbf589065137383492b6dceb413c085d4711d87a744256be3f0f3"} err="failed to get container status \"f217a4c5ad5fbf589065137383492b6dceb413c085d4711d87a744256be3f0f3\": rpc error: code = NotFound desc = could not find container \"f217a4c5ad5fbf589065137383492b6dceb413c085d4711d87a744256be3f0f3\": container with ID starting with f217a4c5ad5fbf589065137383492b6dceb413c085d4711d87a744256be3f0f3 not found: ID does not exist"
Jul 28 03:02:26.233152 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:26.233118    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"]
Jul 28 03:02:26.236039 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:26.236013    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t7lpr"]
Jul 28 03:02:26.986413 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:26.986381    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71b7272c-7b24-4158-8418-f596c7b04364" path="/var/lib/kubelet/pods/71b7272c-7b24-4158-8418-f596c7b04364/volumes"
Jul 28 03:02:27.645540 ip-10-0-139-81 kubenswrapper[2578]: E0728 03:02:27.645492    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 03:02:28.215929 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:02:28.215895    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 03:04:31.216806 ip-10-0-139-81 kubenswrapper[2578]: E0728 03:04:31.216768    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 03:04:31.723838 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:04:31.723800    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 03:06:34.724953 ip-10-0-139-81 kubenswrapper[2578]: E0728 03:06:34.724859    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 03:06:35.230453 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:06:35.230424    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 03:08:29.992731 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:29.992692    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq_55a643be-4bcf-4e39-8705-a5b6e8686d81/istio-proxy/0.log"
Jul 28 03:08:30.022685 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:30.022661    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq_55a643be-4bcf-4e39-8705-a5b6e8686d81/istio-proxy/1.log"
Jul 28 03:08:30.035627 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:30.035603    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68787665fc-c2vr7_3f8169c2-427b-49b7-a2cc-8e8844f0d80c/router/0.log"
Jul 28 03:08:30.933053 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:30.933024    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq_55a643be-4bcf-4e39-8705-a5b6e8686d81/istio-proxy/0.log"
Jul 28 03:08:30.964432 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:30.964406    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq_55a643be-4bcf-4e39-8705-a5b6e8686d81/istio-proxy/1.log"
Jul 28 03:08:30.979407 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:30.979379    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68787665fc-c2vr7_3f8169c2-427b-49b7-a2cc-8e8844f0d80c/router/0.log"
Jul 28 03:08:31.875056 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:31.875023    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq_55a643be-4bcf-4e39-8705-a5b6e8686d81/istio-proxy/0.log"
Jul 28 03:08:31.906226 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:31.906202    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq_55a643be-4bcf-4e39-8705-a5b6e8686d81/istio-proxy/1.log"
Jul 28 03:08:31.922577 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:31.922555    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68787665fc-c2vr7_3f8169c2-427b-49b7-a2cc-8e8844f0d80c/router/0.log"
Jul 28 03:08:32.842975 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:32.842945    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq_55a643be-4bcf-4e39-8705-a5b6e8686d81/istio-proxy/0.log"
Jul 28 03:08:32.871544 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:32.871518    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq_55a643be-4bcf-4e39-8705-a5b6e8686d81/istio-proxy/1.log"
Jul 28 03:08:32.884346 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:32.884312    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68787665fc-c2vr7_3f8169c2-427b-49b7-a2cc-8e8844f0d80c/router/0.log"
Jul 28 03:08:33.719317 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:33.719285    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:34.951057 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:34.951032    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:36.210819 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:36.210790    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:37.287896 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:37.287867    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:38.231600 ip-10-0-139-81 kubenswrapper[2578]: E0728 03:08:38.231538    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="1579c6d1-9645-4acd-b180-a7e0becfe665"
Jul 28 03:08:38.349607 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:38.349578    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:38.745927 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:38.745900    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 03:08:39.443767 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:39.443742    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:40.543307 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:40.543280    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:41.604319 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:41.604290    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:42.672220 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:42.672191    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:43.767913 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:43.767888    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:44.876326 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:44.876292    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:45.968127 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:45.968098    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:47.084654 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:47.084629    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:48.189301 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:48.189275    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:49.271924 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:49.271895    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:50.383641 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:50.383615    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:51.464903 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:51.464875    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:52.549619 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:52.549592    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:53.616979 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:53.616952    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:54.689346 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:54.689311    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:55.762935 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:55.762909    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:56.828489 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:56.828459    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:57.904508 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:57.904480    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:08:59.015848 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:08:59.015821    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:09:00.113484 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:00.113456    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:09:05.007891 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:05.007860    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wpcdj_cd1a9e35-4116-41f7-b275-9da51eb2549a/global-pull-secret-syncer/0.log"
Jul 28 03:09:05.113353 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:05.113318    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c9ntv_069b6671-6207-471b-a09c-26a0293ba1ba/konnectivity-agent/0.log"
Jul 28 03:09:05.203460 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:05.203399    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-81.ec2.internal_216527ecc9658544f9bc7b3b7800c13c/haproxy/0.log"
Jul 28 03:09:10.264018 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:10.263986    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-2w4pm_2677b484-e75e-438a-82ad-7305228b0b80/kuadrant-console-plugin/0.log"
Jul 28 03:09:12.205952 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:12.205928    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jlpsc_fd3e8c04-97b7-4b06-9fe9-d889ac1202ce/node-exporter/0.log"
Jul 28 03:09:12.227134 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:12.227100    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jlpsc_fd3e8c04-97b7-4b06-9fe9-d889ac1202ce/kube-rbac-proxy/0.log"
Jul 28 03:09:12.248696 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:12.248681    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jlpsc_fd3e8c04-97b7-4b06-9fe9-d889ac1202ce/init-textfile/0.log"
Jul 28 03:09:15.046011 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.045980    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"]
Jul 28 03:09:15.046431 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.046269    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b7272c-7b24-4158-8418-f596c7b04364" containerName="main"
Jul 28 03:09:15.046431 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.046281    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b7272c-7b24-4158-8418-f596c7b04364" containerName="main"
Jul 28 03:09:15.046431 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.046305    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47abb7ef-b2ac-47d2-ac98-0820240c57f8" containerName="main"
Jul 28 03:09:15.046431 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.046311    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="47abb7ef-b2ac-47d2-ac98-0820240c57f8" containerName="main"
Jul 28 03:09:15.046431 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.046388    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="47abb7ef-b2ac-47d2-ac98-0820240c57f8" containerName="main"
Jul 28 03:09:15.046431 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.046398    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b7272c-7b24-4158-8418-f596c7b04364" containerName="main"
Jul 28 03:09:15.049159 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.049139    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"
Jul 28 03:09:15.051810 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.051786    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6q4dh\"/\"kube-root-ca.crt\""
Jul 28 03:09:15.053118 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.053102    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6q4dh\"/\"openshift-service-ca.crt\""
Jul 28 03:09:15.053190 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.053143    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6q4dh\"/\"default-dockercfg-fvqfh\""
Jul 28 03:09:15.057859 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.057835    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"]
Jul 28 03:09:15.174577 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.174549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8fcc3e9-cc7f-4280-bbe3-142bab9ba018-sys\") pod \"perf-node-gather-daemonset-gtm8t\" (UID: \"c8fcc3e9-cc7f-4280-bbe3-142bab9ba018\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"
Jul 28 03:09:15.174714 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.174580    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prnrn\" (UniqueName: \"kubernetes.io/projected/c8fcc3e9-cc7f-4280-bbe3-142bab9ba018-kube-api-access-prnrn\") pod \"perf-node-gather-daemonset-gtm8t\" (UID: \"c8fcc3e9-cc7f-4280-bbe3-142bab9ba018\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"
Jul 28 03:09:15.174714 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.174614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c8fcc3e9-cc7f-4280-bbe3-142bab9ba018-podres\") pod \"perf-node-gather-daemonset-gtm8t\" (UID: \"c8fcc3e9-cc7f-4280-bbe3-142bab9ba018\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"
Jul 28 03:09:15.174714 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.174693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c8fcc3e9-cc7f-4280-bbe3-142bab9ba018-proc\") pod \"perf-node-gather-daemonset-gtm8t\" (UID: \"c8fcc3e9-cc7f-4280-bbe3-142bab9ba018\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"
Jul 28 03:09:15.174834 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.174760    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8fcc3e9-cc7f-4280-bbe3-142bab9ba018-lib-modules\") pod \"perf-node-gather-daemonset-gtm8t\" (UID: \"c8fcc3e9-cc7f-4280-bbe3-142bab9ba018\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"
Jul 28 03:09:15.275601 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.275573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c8fcc3e9-cc7f-4280-bbe3-142bab9ba018-proc\") pod \"perf-node-gather-daemonset-gtm8t\" (UID: \"c8fcc3e9-cc7f-4280-bbe3-142bab9ba018\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"
Jul 28 03:09:15.275716 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.275622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8fcc3e9-cc7f-4280-bbe3-142bab9ba018-lib-modules\") pod \"perf-node-gather-daemonset-gtm8t\" (UID: \"c8fcc3e9-cc7f-4280-bbe3-142bab9ba018\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"
Jul 28 03:09:15.275716 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.275657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8fcc3e9-cc7f-4280-bbe3-142bab9ba018-sys\") pod \"perf-node-gather-daemonset-gtm8t\" (UID: \"c8fcc3e9-cc7f-4280-bbe3-142bab9ba018\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"
Jul 28 03:09:15.275716 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.275684    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prnrn\" (UniqueName: \"kubernetes.io/projected/c8fcc3e9-cc7f-4280-bbe3-142bab9ba018-kube-api-access-prnrn\") pod \"perf-node-gather-daemonset-gtm8t\" (UID: \"c8fcc3e9-cc7f-4280-bbe3-142bab9ba018\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"
Jul 28 03:09:15.275716 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.275691    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c8fcc3e9-cc7f-4280-bbe3-142bab9ba018-proc\") pod \"perf-node-gather-daemonset-gtm8t\" (UID: \"c8fcc3e9-cc7f-4280-bbe3-142bab9ba018\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"
Jul 28 03:09:15.275865 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.275717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c8fcc3e9-cc7f-4280-bbe3-142bab9ba018-podres\") pod \"perf-node-gather-daemonset-gtm8t\" (UID: \"c8fcc3e9-cc7f-4280-bbe3-142bab9ba018\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"
Jul 28 03:09:15.275865 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.275752    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8fcc3e9-cc7f-4280-bbe3-142bab9ba018-sys\") pod \"perf-node-gather-daemonset-gtm8t\" (UID: \"c8fcc3e9-cc7f-4280-bbe3-142bab9ba018\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"
Jul 28 03:09:15.275865 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.275819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c8fcc3e9-cc7f-4280-bbe3-142bab9ba018-podres\") pod \"perf-node-gather-daemonset-gtm8t\" (UID: \"c8fcc3e9-cc7f-4280-bbe3-142bab9ba018\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"
Jul 28 03:09:15.275865 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.275826    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8fcc3e9-cc7f-4280-bbe3-142bab9ba018-lib-modules\") pod \"perf-node-gather-daemonset-gtm8t\" (UID: \"c8fcc3e9-cc7f-4280-bbe3-142bab9ba018\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"
Jul 28 03:09:15.284049 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.284026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prnrn\" (UniqueName: \"kubernetes.io/projected/c8fcc3e9-cc7f-4280-bbe3-142bab9ba018-kube-api-access-prnrn\") pod \"perf-node-gather-daemonset-gtm8t\" (UID: \"c8fcc3e9-cc7f-4280-bbe3-142bab9ba018\") " pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"
Jul 28 03:09:15.359487 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.359431    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"
Jul 28 03:09:15.483512 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.483477    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"]
Jul 28 03:09:15.486417 ip-10-0-139-81 kubenswrapper[2578]: W0728 03:09:15.486389    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc8fcc3e9_cc7f_4280_bbe3_142bab9ba018.slice/crio-621130e9c8f3776db3655b46c753551b16e540fb269a3502373d47fa35ea3a26 WatchSource:0}: Error finding container 621130e9c8f3776db3655b46c753551b16e540fb269a3502373d47fa35ea3a26: Status 404 returned error can't find the container with id 621130e9c8f3776db3655b46c753551b16e540fb269a3502373d47fa35ea3a26
Jul 28 03:09:15.912785 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.912750    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t" event={"ID":"c8fcc3e9-cc7f-4280-bbe3-142bab9ba018","Type":"ContainerStarted","Data":"a03fce3e6f1e1fb8ab092d503efc9bbf49eb88cd43c50122d03216e4f16a86c5"}
Jul 28 03:09:15.912944 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.912801    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t" event={"ID":"c8fcc3e9-cc7f-4280-bbe3-142bab9ba018","Type":"ContainerStarted","Data":"621130e9c8f3776db3655b46c753551b16e540fb269a3502373d47fa35ea3a26"}
Jul 28 03:09:15.912944 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.912879    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"
Jul 28 03:09:15.928202 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:15.928160    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t" podStartSLOduration=0.928148351 podStartE2EDuration="928.148351ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 03:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 03:09:15.926501735 +0000 UTC m=+6199.503959365" watchObservedRunningTime="2026-07-28 03:09:15.928148351 +0000 UTC m=+6199.505605967"
Jul 28 03:09:16.979737 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:16.979703    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t5gxc_b6862e7b-5460-406a-b6dc-18de72afce4f/dns/0.log"
Jul 28 03:09:17.001945 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:17.001918    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t5gxc_b6862e7b-5460-406a-b6dc-18de72afce4f/kube-rbac-proxy/0.log"
Jul 28 03:09:17.144599 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:17.144578    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wlzmw_640c683a-7ed6-4686-a0de-f6c631e1ab3a/dns-node-resolver/0.log"
Jul 28 03:09:17.742306 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:17.742278    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2d5z9_b8193a25-2add-42b3-8815-022a163695fe/node-ca/0.log"
Jul 28 03:09:18.950357 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:18.950261    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq_55a643be-4bcf-4e39-8705-a5b6e8686d81/istio-proxy/0.log"
Jul 28 03:09:18.982965 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:18.982866    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq_55a643be-4bcf-4e39-8705-a5b6e8686d81/istio-proxy/1.log"
Jul 28 03:09:19.004880 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:19.004855    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68787665fc-c2vr7_3f8169c2-427b-49b7-a2cc-8e8844f0d80c/router/0.log"
Jul 28 03:09:19.528276 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:19.528250    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ls4sg_4678fb71-04c1-4342-aee2-34eb7ac2c129/serve-healthcheck-canary/0.log"
Jul 28 03:09:20.177162 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:20.177132    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mxrqp_5ac739c6-6fe9-43a1-a92d-e1298aea7774/kube-rbac-proxy/0.log"
Jul 28 03:09:20.198150 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:20.198126    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mxrqp_5ac739c6-6fe9-43a1-a92d-e1298aea7774/exporter/0.log"
Jul 28 03:09:20.220008 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:20.219991    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mxrqp_5ac739c6-6fe9-43a1-a92d-e1298aea7774/extractor/0.log"
Jul 28 03:09:21.925997 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:21.925971    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6q4dh/perf-node-gather-daemonset-gtm8t"
Jul 28 03:09:23.060981 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:23.060954    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-78db6c7dfd-gspjc_f1ea1355-2483-4f56-a5b2-f4967d4199a0/manager/0.log"
Jul 28 03:09:23.123847 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:23.123825    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-m8rzf_c2d8b62d-c846-4e91-8273-25cffbd7040a/openshift-lws-operator/0.log"
Jul 28 03:09:24.721150 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:24.721121    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-rtfzg_40c6c649-9390-475e-a84e-b95b541fb6f0/seaweedfs/0.log"
Jul 28 03:09:24.969852 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:24.969821    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq_55a643be-4bcf-4e39-8705-a5b6e8686d81/istio-proxy/0.log"
Jul 28 03:09:25.002263 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:25.002240    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq_55a643be-4bcf-4e39-8705-a5b6e8686d81/istio-proxy/1.log"
Jul 28 03:09:25.018665 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:25.018635    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68787665fc-c2vr7_3f8169c2-427b-49b7-a2cc-8e8844f0d80c/router/0.log"
Jul 28 03:09:26.306931 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:26.306906    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq_55a643be-4bcf-4e39-8705-a5b6e8686d81/istio-proxy/0.log"
Jul 28 03:09:26.336177 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:26.336153    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-cx6zq_55a643be-4bcf-4e39-8705-a5b6e8686d81/istio-proxy/1.log"
Jul 28 03:09:26.363879 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:26.363839    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68787665fc-c2vr7_3f8169c2-427b-49b7-a2cc-8e8844f0d80c/router/0.log"
Jul 28 03:09:29.739104 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:29.739075    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-rkmdz_b4d9fecf-bea9-4cd4-977a-ae015ee24e38/migrator/0.log"
Jul 28 03:09:29.760186 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:29.760161    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-rkmdz_b4d9fecf-bea9-4cd4-977a-ae015ee24e38/graceful-termination/0.log"
Jul 28 03:09:30.214756 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:30.214724    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2_79ec8bac-483f-4773-a8b0-5ac56d23c8bb/kube-storage-version-migrator-operator/1.log"
Jul 28 03:09:30.215730 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:30.215710    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-tk5p2_79ec8bac-483f-4773-a8b0-5ac56d23c8bb/kube-storage-version-migrator-operator/0.log"
Jul 28 03:09:31.344133 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:31.344049    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gwwvx_69bc94d5-8275-4e49-b94f-03ef28a1e2b2/kube-multus-additional-cni-plugins/0.log"
Jul 28 03:09:31.379308 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:31.379277    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gwwvx_69bc94d5-8275-4e49-b94f-03ef28a1e2b2/egress-router-binary-copy/0.log"
Jul 28 03:09:31.414720 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:31.414696    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gwwvx_69bc94d5-8275-4e49-b94f-03ef28a1e2b2/cni-plugins/0.log"
Jul 28 03:09:31.456856 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:31.456829    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gwwvx_69bc94d5-8275-4e49-b94f-03ef28a1e2b2/bond-cni-plugin/0.log"
Jul 28 03:09:31.495916 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:31.495894    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gwwvx_69bc94d5-8275-4e49-b94f-03ef28a1e2b2/routeoverride-cni/0.log"
Jul 28 03:09:31.530792 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:31.530765    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gwwvx_69bc94d5-8275-4e49-b94f-03ef28a1e2b2/whereabouts-cni-bincopy/0.log"
Jul 28 03:09:31.565129 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:31.565105    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gwwvx_69bc94d5-8275-4e49-b94f-03ef28a1e2b2/whereabouts-cni/0.log"
Jul 28 03:09:31.985962 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:31.985938    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qqq2z_65e2f293-2ed4-4ff1-aba1-ca582bde2904/kube-multus/0.log"
Jul 28 03:09:32.094674 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:32.094647    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dx5sm_ddcb15a3-037d-45a0-8787-12585c0d2465/network-metrics-daemon/0.log"
Jul 28 03:09:32.128943 ip-10-0-139-81 kubenswrapper[2578]: I0728 03:09:32.128915    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dx5sm_ddcb15a3-037d-45a0-8787-12585c0d2465/kube-rbac-proxy/0.log"