Jul 27 17:00:54.272470 ip-10-0-143-86 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 17:00:54.711302 ip-10-0-143-86 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 17:00:54.711302 ip-10-0-143-86 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 17:00:54.711302 ip-10-0-143-86 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 17:00:54.711302 ip-10-0-143-86 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 17:00:54.711302 ip-10-0-143-86 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 17:00:54.712409 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.712323    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 17:00:54.718979 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.718963    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:00:54.718979 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.718978    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.718982    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.718986    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.718989    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.718992    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.718995    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.718998    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719001    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719004    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719007    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719009    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719013    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719016    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719019    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719021    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719024    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719027    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719031    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719033    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719036    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:00:54.719044 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719039    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:00:54.719528 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719042    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:00:54.719528 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719045    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:00:54.719528 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719048    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:00:54.719528 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719050    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:00:54.719528 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719053    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:00:54.719528 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719056    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:00:54.719528 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719059    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:00:54.719528 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719061    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:00:54.719528 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719066    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:00:54.719528 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719070    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:00:54.719528 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719073    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:00:54.719528 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719077    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:00:54.719528 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719080    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:00:54.719528 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719083    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:00:54.719528 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719086    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:00:54.719528 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719088    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:00:54.719528 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719091    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:00:54.719528 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719093    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:00:54.719528 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719096    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719098    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719101    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719103    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719106    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719110    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719114    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719116    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719119    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719121    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719123    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719127    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719129    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719132    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719134    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719139    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719142    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719144    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719147    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719149    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:00:54.719992 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719152    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:00:54.720490 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719155    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:00:54.720490 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719158    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:00:54.720490 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719161    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:00:54.720490 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719163    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:00:54.720490 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719175    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:00:54.720490 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719178    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:00:54.720490 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719180    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:00:54.720490 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719183    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:00:54.720490 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719186    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:00:54.720490 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719189    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:00:54.720490 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719193    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:00:54.720490 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719195    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:00:54.720490 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719198    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:00:54.720490 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719201    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:00:54.720490 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719203    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:00:54.720490 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719206    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:00:54.720490 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719209    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:00:54.720490 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719211    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:00:54.720490 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719214    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719217    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719219    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719222    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719225    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719227    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719230    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719233    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719236    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719241    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719244    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719247    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719250    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719253    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719256    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719259    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719261    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719264    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719267    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719270    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:00:54.720943 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719963    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719969    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719972    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719975    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719978    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719981    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719984    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719986    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719990    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719993    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719996    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.719999    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720002    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720006    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720009    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720012    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720014    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720017    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720020    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720023    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:00:54.721492 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720026    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720029    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720032    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720035    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720037    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720040    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720043    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720046    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720048    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720051    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720053    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720056    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720059    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720061    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720063    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720066    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720069    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720071    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720073    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720076    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:00:54.721996 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720078    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:00:54.722497 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720081    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:00:54.722497 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720084    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:00:54.722497 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720088    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:00:54.722497 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720091    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:00:54.722497 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720093    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:00:54.722497 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720096    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:00:54.722497 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720099    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:00:54.722497 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720101    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:00:54.722497 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720104    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:00:54.722497 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720106    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:00:54.722497 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720109    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:00:54.722497 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720113    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:00:54.722497 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720116    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:00:54.722497 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720118    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:00:54.722497 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720121    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:00:54.722497 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720123    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:00:54.722497 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720126    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:00:54.722497 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720129    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:00:54.722497 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720131    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:00:54.722976 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720134    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:00:54.722976 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720136    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:00:54.722976 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720139    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:00:54.722976 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720144    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:00:54.722976 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720147    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:00:54.722976 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720151    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:00:54.722976 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720155    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:00:54.722976 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720158    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:00:54.722976 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720161    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:00:54.722976 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720164    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:00:54.722976 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720180    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:00:54.722976 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720182    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:00:54.722976 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720185    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:00:54.722976 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720188    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:00:54.722976 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720191    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:00:54.722976 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720193    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:00:54.722976 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720196    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:00:54.722976 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720198    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:00:54.722976 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720201    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720203    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720206    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720208    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720211    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720213    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720218    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720221    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720224    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720226    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720229    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720231    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720234    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720236    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720239    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720241    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720244    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720246    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720249    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720251    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:00:54.723464 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720254    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:00:54.723978 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720609    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:00:54.723978 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720614    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:00:54.723978 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720616    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:00:54.723978 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720619    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:00:54.723978 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720622    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:00:54.723978 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720624    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:00:54.723978 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720627    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:00:54.723978 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720630    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:00:54.723978 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720632    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:00:54.723978 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720635    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:00:54.723978 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720637    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:00:54.723978 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720640    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:00:54.723978 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720642    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:00:54.723978 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720645    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:00:54.723978 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720647    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:00:54.723978 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720649    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:00:54.723978 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720652    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:00:54.723978 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720656    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:00:54.723978 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720659    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720661    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720664    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720667    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720669    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720672    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720674    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720677    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720679    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720682    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720685    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720687    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720690    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720692    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720695    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720697    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720700    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720703    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720705    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720708    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:00:54.724454 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720710    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720713    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720716    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720718    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720721    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720723    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720725    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720728    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720730    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720733    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720738    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720741    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720744    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720747    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720750    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720753    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720755    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720758    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720761    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720764    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:00:54.725019 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720766    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720768    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720771    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720773    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720775    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720778    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720780    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720783    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720785    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720788    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720790    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720792    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720795    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720800    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720802    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720805    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720807    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720810    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720813    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720816    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:00:54.725517 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720818    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:00:54.726049 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720821    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:00:54.726049 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720823    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:00:54.726049 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720826    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:00:54.726049 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720829    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:00:54.726049 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720832    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:00:54.726049 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720834    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:00:54.726049 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720837    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:00:54.726049 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720839    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:00:54.726049 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720843    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:00:54.726049 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720847    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:00:54.726049 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720850    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:00:54.726049 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720853    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:00:54.726049 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720855    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:00:54.726049 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720858    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:00:54.726049 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720861    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:00:54.726049 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720863    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:00:54.726049 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720866    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:00:54.726049 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720868    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:00:54.726049 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720871    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.720875    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.720944    2566 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.720951    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.720959    2566 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.720965    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.720973    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.720978    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.720984    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.720989    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.720992    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.720995    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.720998    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721002    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721005    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721008    2566 flags.go:64] FLAG: --cgroup-root=""
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721011    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721014    2566 flags.go:64] FLAG: --client-ca-file=""
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721017    2566 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721020    2566 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721025    2566 flags.go:64] FLAG: --cluster-domain=""
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721028    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721031    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 17:00:54.726538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721034    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721039    2566 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721042    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721045    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721048    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721052    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721055    2566 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721058    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721061    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721064    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721068    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721072    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721075    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721078    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721080    2566 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721085    2566 flags.go:64] FLAG: --enable-server="true"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721087    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721091    2566 flags.go:64] FLAG: --event-burst="100"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721095    2566 flags.go:64] FLAG: --event-qps="50"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721098    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721101    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721104    2566 flags.go:64] FLAG: --eviction-hard=""
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721107    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721110    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721113    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 17:00:54.727079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721116    2566 flags.go:64] FLAG: --eviction-soft=""
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721119    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721122    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721125    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721128    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721131    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721134    2566 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721137    2566 flags.go:64] FLAG: --feature-gates=""
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721142    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721145    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721148    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721151    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721154    2566 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721157    2566 flags.go:64] FLAG: --help="false"
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721159    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-143-86.ec2.internal"
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721162    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721178    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721181    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721185    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721188    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721191    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721194    2566 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721198    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 17:00:54.727689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721201    2566 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721204    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721207    2566 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721210    2566 flags.go:64] FLAG: --kube-reserved=""
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721213    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721216    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721219    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721222    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721224    2566 flags.go:64] FLAG: --lock-file=""
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721228    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721231    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721234    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721239    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721242    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721246    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721249    2566 flags.go:64] FLAG: --logging-format="text"
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721252    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721256    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721259    2566 flags.go:64] FLAG: --manifest-url=""
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721263    2566 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721267    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721270    2566 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721275    2566 flags.go:64] FLAG: --max-pods="110"
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721277    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721280    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 17:00:54.728320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721283    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721286    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721289    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721292    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721295    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721302    2566 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721307    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721310    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721313    2566 flags.go:64] FLAG: --pod-cidr=""
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721315    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721322    2566 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721325    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721329    2566 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721331    2566 flags.go:64] FLAG: --port="10250"
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721335    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721338    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ac0005e1424d1cd7"
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721341    2566 flags.go:64] FLAG: --qos-reserved=""
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721344    2566 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721347    2566 flags.go:64] FLAG: --register-node="true"
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721350    2566 flags.go:64] FLAG: --register-with-taints=""
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721353    2566 flags.go:64] FLAG: --registry-burst="10"
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721356    2566 flags.go:64] FLAG: --registry-qps="5"
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721359    2566 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721362    2566 flags.go:64] FLAG: --reserved-memory=""
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721367    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 17:00:54.729289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721370    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721373    2566 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721376    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721379    2566 flags.go:64] FLAG: --runonce="false"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721382    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721384    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721387    2566 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721390    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721393    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721396    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721399    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721402    2566 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721405    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721407    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721413    2566 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721416    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721419    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721422    2566 flags.go:64] FLAG: --system-cgroups=""
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721424    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721429    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721432    2566 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721435    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721440    2566 flags.go:64] FLAG: --tls-min-version=""
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721442    2566 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721445    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 17:00:54.730180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721448    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721451    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721453    2566 flags.go:64] FLAG: --v="2"
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721458    2566 flags.go:64] FLAG: --version="false"
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721461    2566 flags.go:64] FLAG: --vmodule=""
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721465    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.721470    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721601    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721605    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721608    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721611    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721615    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721618    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721621    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721624    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721627    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721629    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721632    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721635    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721638    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721640    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:00:54.730799 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721645    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:00:54.731328 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721648    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:00:54.731328 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721652    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:00:54.731328 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721655    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:00:54.731328 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721658    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:00:54.731328 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721661    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:00:54.731328 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721665    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:00:54.731328 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721669    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:00:54.731328 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721672    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:00:54.731328 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721674    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:00:54.731328 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721677    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:00:54.731328 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721680    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:00:54.731328 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721683    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:00:54.731328 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721685    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:00:54.731328 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721688    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:00:54.731328 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721691    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:00:54.731328 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721693    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:00:54.731328 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721697    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:00:54.731328 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721700    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:00:54.731328 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721703    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721705    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721707    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721710    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721713    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721715    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721717    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721720    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721723    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721725    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721728    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721730    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721733    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721737    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721740    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721742    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721745    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721747    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721750    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721752    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:00:54.731779 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721755    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:00:54.732292 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721757    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:00:54.732292 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721760    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:00:54.732292 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721762    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:00:54.732292 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721765    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:00:54.732292 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721767    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:00:54.732292 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721770    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:00:54.732292 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721773    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:00:54.732292 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721776    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:00:54.732292 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721778    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:00:54.732292 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721782    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:00:54.732292 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721789    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:00:54.732292 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721791    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:00:54.732292 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721794    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:00:54.732292 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721796    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:00:54.732292 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721799    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:00:54.732292 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721803    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:00:54.732292 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721806    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:00:54.732292 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721809    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:00:54.732292 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721812    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:00:54.732763 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721815    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:00:54.732763 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721817    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:00:54.732763 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721820    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:00:54.732763 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721822    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:00:54.732763 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721824    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:00:54.732763 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721828    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:00:54.732763 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721831    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:00:54.732763 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721834    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:00:54.732763 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721836    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:00:54.732763 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721839    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:00:54.732763 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721842    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:00:54.732763 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721844    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:00:54.732763 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721847    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:00:54.732763 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721849    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:00:54.732763 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721852    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:00:54.732763 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721854    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:00:54.732763 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721857    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:00:54.732763 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721859    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:00:54.732763 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721862    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:00:54.733248 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721864    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:00:54.733248 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721867    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:00:54.733248 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721869    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:00:54.733248 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721873    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:00:54.733248 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721875    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:00:54.733248 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721878    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:00:54.733248 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.721881    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:00:54.733248 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.722604    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 17:00:54.733248 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.731084    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 17:00:54.733248 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.731100    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 17:00:54.733248 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.731990    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 17:00:54.733248 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.732002    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 17:00:54.733248 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732057    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:00:54.733248 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732062    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:00:54.733592 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732066    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:00:54.733592 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732069    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:00:54.733592 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732072    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:00:54.733592 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732075    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:00:54.733592 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732077    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:00:54.733592 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732080    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:00:54.733592 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732083    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:00:54.733592 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732085    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:00:54.733592 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732088    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:00:54.733592 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732091    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:00:54.733592 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732093    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:00:54.733592 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732096    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:00:54.733592 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732099    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:00:54.733592 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732102    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:00:54.733592 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732105    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:00:54.733592 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732108    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:00:54.733592 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732112    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:00:54.733592 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732116    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:00:54.733592 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732119    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732122    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732125    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732129    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732132    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732135    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732137    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732140    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732143    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732146    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732149    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732152    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732154    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732157    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732159    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732163    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732176    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732179    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732181    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732184    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:00:54.734047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732187    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732189    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732192    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732194    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732197    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732200    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732203    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732205    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732208    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732211    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732213    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732216    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732219    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732221    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732224    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732227    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732229    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732232    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732234    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732236    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:00:54.734548 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732239    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732243    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732245    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732248    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732251    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732254    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732256    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732259    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732261    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732264    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732266    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732269    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732271    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732274    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732276    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732281    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732284    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732287    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732290    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732293    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:00:54.735027 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732296    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:00:54.735571 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732299    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:00:54.735571 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732301    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:00:54.735571 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732304    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:00:54.735571 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732307    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:00:54.735571 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732309    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:00:54.735571 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732312    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:00:54.735571 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732315    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:00:54.735571 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732317    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:00:54.735571 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732320    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:00:54.735571 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732322    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:00:54.735571 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732325    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:00:54.735571 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732328    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:00:54.735571 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732331    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:00:54.735571 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732333    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:00:54.735571 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732336    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:00:54.735571 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732338    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:00:54.735571 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732341    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:00:54.735571 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732344    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:00:54.736070 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.732349    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 17:00:54.736070 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732479    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:00:54.736070 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732484    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:00:54.736070 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732487    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:00:54.736070 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732490    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:00:54.736070 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732492    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:00:54.736070 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732495    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:00:54.736070 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732497    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:00:54.736070 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732500    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:00:54.736070 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732503    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:00:54.736070 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732505    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:00:54.736070 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732508    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:00:54.736070 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732510    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:00:54.736070 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732513    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:00:54.736070 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732517    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:00:54.736453 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732521    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:00:54.736453 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732524    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:00:54.736453 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732527    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:00:54.736453 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732530    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:00:54.736453 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732532    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:00:54.736453 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732535    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:00:54.736453 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732539    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:00:54.736453 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732543    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:00:54.736453 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732547    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:00:54.736453 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732550    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:00:54.736453 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732553    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:00:54.736453 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732557    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:00:54.736453 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732560    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:00:54.736453 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732562    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:00:54.736453 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732565    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:00:54.736453 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732567    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:00:54.736453 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732570    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:00:54.736453 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732572    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:00:54.736453 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732575    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732578    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732580    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732583    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732585    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732588    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732590    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732593    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732595    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732598    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732601    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732603    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732606    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732609    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732612    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732614    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732617    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732619    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732622    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732625    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:00:54.736914 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732627    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732630    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732632    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732635    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732637    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732640    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732643    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732645    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732648    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732651    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732653    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732656    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732658    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732661    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732663    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732666    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732668    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732671    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732673    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732675    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:00:54.737431 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732678    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732681    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732683    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732685    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732688    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732690    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732693    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732696    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732699    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732701    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732704    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732706    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732709    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732711    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732714    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732716    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732718    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732721    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732724    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732727    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:00:54.737899 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732729    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:00:54.738396 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732731    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:00:54.738396 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732734    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:00:54.738396 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732736    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:00:54.738396 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732739    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:00:54.738396 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732741    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:00:54.738396 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:54.732744    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:00:54.738396 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.732748    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 17:00:54.738396 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.732846    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 17:00:54.738396 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.735273    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 17:00:54.738396 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.736986    2566 server.go:1019] "Starting client certificate rotation"
Jul 27 17:00:54.738396 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.737089    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 17:00:54.738396 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.737127    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 17:00:54.739747 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.739734    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 17:00:54.740883 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.740867    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 17:00:54.749805 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.749791    2566 log.go:25] "Validated CRI v1 runtime API"
Jul 27 17:00:54.752254 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.752239    2566 log.go:25] "Validated CRI v1 image API"
Jul 27 17:00:54.752970 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.752954    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 17:00:54.755598 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.755577    2566 fs.go:135] Filesystem UUIDs: map[2e103e63-5ed6-4f92-9f7a-acb54978128c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 da8f8864-8e97-4a34-abda-30e66105b042:/dev/nvme0n1p4]
Jul 27 17:00:54.755680 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.755595    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 17:00:54.763397 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.763238    2566 manager.go:216] Machine: {Timestamp:2026-07-27 17:00:54.761491527 +0000 UTC m=+0.362755018 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3122637 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ff1b3b901a8f5a79f6bc6a31b2141 SystemUUID:ec2ff1b3-b901-a8f5-a79f-6bc6a31b2141 BootID:1b950fde-36d7-4db6-baca-3daa953bb03b Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d5:57:bf:fd:31 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d5:57:bf:fd:31 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:02:24:b1:37:b6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 17:00:54.763397 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.763387    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 17:00:54.763542 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.763476    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 17:00:54.767480 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.767451    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 17:00:54.767636 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.767485    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-86.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 17:00:54.768379 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.768367    2566 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 17:00:54.768379 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.768380    2566 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 17:00:54.768464 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.768388    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 17:00:54.769071 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.769060    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 17:00:54.770301 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.770291    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 17:00:54.771580 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.771564    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 17:00:54.773105 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.773094    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 17:00:54.773241 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.773227    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 17:00:54.773295 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.773256    2566 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 17:00:54.773295 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.773266    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 17:00:54.773295 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.773278    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 17:00:54.773295 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.773286    2566 kubelet.go:398] "Adding apiserver pod source"
Jul 27 17:00:54.773295 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.773295    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 17:00:54.773628 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.773617    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 17:00:54.773673 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.773638    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 17:00:54.775878 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.775862    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 17:00:54.778355 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.778341    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 17:00:54.778411 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.778383    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 17:00:54.780473 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.780459    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 17:00:54.780532 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.780477    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 17:00:54.780532 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.780482    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 17:00:54.780532 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.780487    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 17:00:54.780532 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.780496    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 17:00:54.780532 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.780503    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 17:00:54.780532 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.780514    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 17:00:54.780532 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.780522    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 17:00:54.780532 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.780531    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 17:00:54.780744 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.780538    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 17:00:54.780744 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.780546    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 17:00:54.780744 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.780551    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 17:00:54.782215 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.782202    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 17:00:54.782215 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.782214    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 17:00:54.783833 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:54.783812    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-86.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 17:00:54.783901 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.783860    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-86.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 17:00:54.783943 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:54.783895    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 17:00:54.785237 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.785223    2566 server.go:1268] "Started kubelet"
Jul 27 17:00:54.785326 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.785299    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 17:00:54.785387 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.785305    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 17:00:54.785387 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.785377    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 17:00:54.786002 ip-10-0-143-86 systemd[1]: Started Kubernetes Kubelet.
Jul 27 17:00:54.787480 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.787461    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 17:00:54.787831 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.787726    2566 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 17:00:54.790994 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:54.790973    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 17:00:54.796933 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.796917    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 17:00:54.796933 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.796924    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 17:00:54.797131 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:54.797101    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-86.ec2.internal\" not found"
Jul 27 17:00:54.797675 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.797660    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 17:00:54.797752 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.797741    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 17:00:54.797812 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.797659    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 17:00:54.797812 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.797789    2566 factory.go:55] Registering systemd factory
Jul 27 17:00:54.797812 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.797803    2566 factory.go:223] Registration of the systemd container factory successfully
Jul 27 17:00:54.797935 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.797876    2566 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 17:00:54.797935 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.797887    2566 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 17:00:54.798041 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.798023    2566 factory.go:153] Registering CRI-O factory
Jul 27 17:00:54.798041 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.798042    2566 factory.go:223] Registration of the crio container factory successfully
Jul 27 17:00:54.798218 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.798089    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 17:00:54.798218 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.798111    2566 factory.go:103] Registering Raw factory
Jul 27 17:00:54.798218 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.798127    2566 manager.go:1196] Started watching for new ooms in manager
Jul 27 17:00:54.798594 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.798579    2566 manager.go:349] Starting recovery of all containers
Jul 27 17:00:54.800257 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:54.800236    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-86.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 17:00:54.800358 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:54.800341    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 17:00:54.803227 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:54.800286    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-86.ec2.internal.18c634296a91610f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-86.ec2.internal,UID:ip-10-0-143-86.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-86.ec2.internal,},FirstTimestamp:2026-07-27 17:00:54.785196303 +0000 UTC m=+0.386459781,LastTimestamp:2026-07-27 17:00:54.785196303 +0000 UTC m=+0.386459781,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-86.ec2.internal,}"
Jul 27 17:00:54.810781 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.810748    2566 manager.go:354] Recovery completed
Jul 27 17:00:54.815270 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.815258    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:00:54.816966 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.816943    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:00:54.817026 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.816977    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:00:54.817026 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.816987    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:00:54.817416 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.817404    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 17:00:54.817416 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.817415    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 17:00:54.817503 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.817429    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 17:00:54.817503 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.817415    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:00:54.818126 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.818113    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:00:54.818211 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.818137    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:00:54.818211 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.818154    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:00:54.818866 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.818848    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q5fqn"
Jul 27 17:00:54.818981 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:54.818917    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-86.ec2.internal.18c634296c762541  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-86.ec2.internal,UID:ip-10-0-143-86.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-86.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-86.ec2.internal,},FirstTimestamp:2026-07-27 17:00:54.816965953 +0000 UTC m=+0.418229431,LastTimestamp:2026-07-27 17:00:54.816965953 +0000 UTC m=+0.418229431,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-86.ec2.internal,}"
Jul 27 17:00:54.820196 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.820182    2566 policy_none.go:49] "None policy: Start"
Jul 27 17:00:54.820268 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.820201    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 17:00:54.820268 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.820214    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 17:00:54.822089 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.822076    2566 policy_none.go:47] "Start"
Jul 27 17:00:54.827607 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.827587    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q5fqn"
Jul 27 17:00:54.829673 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:54.829612    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-86.ec2.internal.18c634296c766592  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-86.ec2.internal,UID:ip-10-0-143-86.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-86.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-86.ec2.internal,},FirstTimestamp:2026-07-27 17:00:54.816982418 +0000 UTC m=+0.418245895,LastTimestamp:2026-07-27 17:00:54.816982418 +0000 UTC m=+0.418245895,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-86.ec2.internal,}"
Jul 27 17:00:54.847761 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.847746    2566 manager.go:337] "Starting Device Plugin manager"
Jul 27 17:00:54.847819 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:54.847779    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 17:00:54.847819 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.847791    2566 server.go:87] "Starting device plugin registration server"
Jul 27 17:00:54.848023 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.848010    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 17:00:54.848082 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.848024    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 17:00:54.848142 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.848114    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 17:00:54.848206 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.848196    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 17:00:54.848206 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.848206    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 17:00:54.848812 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:54.848791    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 17:00:54.848889 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:54.848848    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-86.ec2.internal\" not found"
Jul 27 17:00:54.879882 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.869378    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 17:00:54.879882 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.870532    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 17:00:54.879882 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.870586    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 17:00:54.879882 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.870610    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 17:00:54.879882 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:54.870650    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 17:00:54.879882 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.872348    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:00:54.948428 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.948409    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:00:54.949449 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.949434    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:00:54.949516 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.949464    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:00:54.949516 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.949474    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:00:54.949516 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.949494    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-86.ec2.internal"
Jul 27 17:00:54.958351 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.958337    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-86.ec2.internal"
Jul 27 17:00:54.958413 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:54.958360    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-86.ec2.internal\": node \"ip-10-0-143-86.ec2.internal\" not found"
Jul 27 17:00:54.971572 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.971520    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal"]
Jul 27 17:00:54.971659 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.971582    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:00:54.973940 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.973926    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:00:54.974009 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.973950    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:00:54.974009 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.973959    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:00:54.976273 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.976261    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:00:54.976410 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.976394    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal"
Jul 27 17:00:54.976442 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.976426    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:00:54.976904 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.976888    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:00:54.977000 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.976907    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:00:54.977000 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.976914    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:00:54.977000 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.976924    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:00:54.977000 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.976928    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:00:54.977000 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.976944    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:00:54.979102 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.979086    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal"
Jul 27 17:00:54.979185 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.979109    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:00:54.979710 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.979697    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:00:54.979776 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.979724    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:00:54.979776 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.979738    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:00:54.989922 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:54.989908    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-86.ec2.internal\" not found"
Jul 27 17:00:54.999078 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.999061    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02541bc35c7e12918e006d46c38418ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal\" (UID: \"02541bc35c7e12918e006d46c38418ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal"
Jul 27 17:00:54.999141 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.999094    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02541bc35c7e12918e006d46c38418ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal\" (UID: \"02541bc35c7e12918e006d46c38418ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal"
Jul 27 17:00:54.999141 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:54.999111    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/81d1b794894ec66f164d640b4bfc3b9a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-86.ec2.internal\" (UID: \"81d1b794894ec66f164d640b4bfc3b9a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal"
Jul 27 17:00:55.017761 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:55.017737    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-86.ec2.internal\" not found" node="ip-10-0-143-86.ec2.internal"
Jul 27 17:00:55.022032 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:55.022018    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-86.ec2.internal\" not found" node="ip-10-0-143-86.ec2.internal"
Jul 27 17:00:55.090163 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:55.090144    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-86.ec2.internal\" not found"
Jul 27 17:00:55.099982 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:55.099963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/81d1b794894ec66f164d640b4bfc3b9a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-86.ec2.internal\" (UID: \"81d1b794894ec66f164d640b4bfc3b9a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal"
Jul 27 17:00:55.100036 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:55.099987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02541bc35c7e12918e006d46c38418ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal\" (UID: \"02541bc35c7e12918e006d46c38418ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal"
Jul 27 17:00:55.100036 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:55.100003    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02541bc35c7e12918e006d46c38418ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal\" (UID: \"02541bc35c7e12918e006d46c38418ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal"
Jul 27 17:00:55.100102 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:55.100048    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/81d1b794894ec66f164d640b4bfc3b9a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-86.ec2.internal\" (UID: \"81d1b794894ec66f164d640b4bfc3b9a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal"
Jul 27 17:00:55.100102 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:55.100074    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02541bc35c7e12918e006d46c38418ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal\" (UID: \"02541bc35c7e12918e006d46c38418ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal"
Jul 27 17:00:55.100102 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:55.100049    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02541bc35c7e12918e006d46c38418ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal\" (UID: \"02541bc35c7e12918e006d46c38418ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal"
Jul 27 17:00:55.190240 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:55.190203    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-86.ec2.internal\" not found"
Jul 27 17:00:55.291049 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:55.290974    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-86.ec2.internal\" not found"
Jul 27 17:00:55.321454 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:55.321435    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal"
Jul 27 17:00:55.327209 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:55.327159    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal"
Jul 27 17:00:55.392065 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:55.392035    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-86.ec2.internal\" not found"
Jul 27 17:00:57.018533 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:57.018516    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-86.ec2.internal\" not found"
Jul 27 17:00:57.119117 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:57.119049    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-86.ec2.internal\" not found"
Jul 27 17:00:57.179986 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:57.179760    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:00:57.219804 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:57.219768    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-86.ec2.internal\" not found"
Jul 27 17:00:57.262222 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:57.262204    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 17:00:57.262654 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:57.262312    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 17:00:57.262654 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:57.262371    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 17:00:57.320634 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:57.320604    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-86.ec2.internal\" not found"
Jul 27 17:00:57.360226 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:57.360194    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 16:55:56 +0000 UTC" deadline="2028-01-26 02:30:00.617993062 +0000 UTC"
Jul 27 17:00:57.360226 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:57.360223    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13137h29m3.257773692s"
Jul 27 17:00:57.410576 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:57.410546    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:00:57.420929 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:57.420903    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-86.ec2.internal\" not found"
Jul 27 17:00:57.496385 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:57.496364    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:00:57.497411 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:57.497393    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:00:57.497519 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:57.497428    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:00:57.497519 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:57.497443    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:00:57.501207 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:57.501181    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d1b794894ec66f164d640b4bfc3b9a.slice/crio-0e9f5b5050cd766cacac78522d5fd1b1a45c1202db639e210eab00812500889e WatchSource:0}: Error finding container 0e9f5b5050cd766cacac78522d5fd1b1a45c1202db639e210eab00812500889e: Status 404 returned error can't find the container with id 0e9f5b5050cd766cacac78522d5fd1b1a45c1202db639e210eab00812500889e
Jul 27 17:00:57.501723 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:57.501705    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02541bc35c7e12918e006d46c38418ad.slice/crio-4f16efa4e40e1ad4e95637ecdbf7e70a0ab4a8ac57f22094358b83cf97826378 WatchSource:0}: Error finding container 4f16efa4e40e1ad4e95637ecdbf7e70a0ab4a8ac57f22094358b83cf97826378: Status 404 returned error can't find the container with id 4f16efa4e40e1ad4e95637ecdbf7e70a0ab4a8ac57f22094358b83cf97826378
Jul 27 17:00:57.505944 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:57.505930    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:00:57.521204 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:57.521190    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-86.ec2.internal\" not found"
Jul 27 17:00:57.621859 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:57.621827    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-86.ec2.internal\" not found"
Jul 27 17:00:57.722301 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:57.722282    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-86.ec2.internal\" not found"
Jul 27 17:00:57.775209 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:57.775192    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:00:57.823113 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:57.823089    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal"
Jul 27 17:00:57.848784 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:57.848762    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 17:00:57.849913 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:57.849900    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal"
Jul 27 17:00:57.861533 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:57.861517    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 17:00:58.101189 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.101100    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:00:58.299726 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.299691    2566 apiserver.go:52] "Watching apiserver"
Jul 27 17:00:58.308063 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.307885    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 17:00:58.308390 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.308359    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-9sh7d","kube-system/global-pull-secret-syncer-8jgtw","kube-system/konnectivity-agent-vclxn","kube-system/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk","openshift-cluster-node-tuning-operator/tuned-vh2nd","openshift-dns/node-resolver-6kcft","openshift-multus/network-metrics-daemon-d72jm","openshift-ovn-kubernetes/ovnkube-node-m9k9v","openshift-image-registry/node-ca-2hn7j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal","openshift-multus/multus-8m5b9","openshift-multus/multus-additional-cni-plugins-2mjl8","openshift-network-diagnostics/network-check-target-6pt78"]
Jul 27 17:00:58.310956 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.310934    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2hn7j"
Jul 27 17:00:58.313268 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.313246    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.313661 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.313569    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tjst9\""
Jul 27 17:00:58.313767 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.313736    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 17:00:58.313868 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.313846    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 17:00:58.313950 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.313904    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 17:00:58.315461 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.315431    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vclxn"
Jul 27 17:00:58.315557 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.315476    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 17:00:58.316021 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.315784    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r2twh\""
Jul 27 17:00:58.316021 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.315869    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 17:00:58.316021 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.315949    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 17:00:58.316021 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.315997    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 17:00:58.316237 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.315994    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 17:00:58.317912 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.317527    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 17:00:58.318008 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.317937    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 17:00:58.318224 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.318198    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-krnrc\""
Jul 27 17:00:58.318580 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.318200    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:00:58.318580 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:58.318393    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pt78" podUID="0ab2a8bd-0d12-41c0-ae51-b4e3b262a896"
Jul 27 17:00:58.322026 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.320396    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.323137 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.323109    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 17:00:58.324726 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.323653    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 17:00:58.324726 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.323830    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ltmlm\""
Jul 27 17:00:58.324726 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.323882    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 17:00:58.324726 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.323924    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 17:00:58.324726 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.324158    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.326515 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.326357    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-d6n57\""
Jul 27 17:00:58.326515 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.326392    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:00:58.326515 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.326458    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 17:00:58.326981 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.326964    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.329096 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.329078    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 17:00:58.329259 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.329241    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hdt6r\""
Jul 27 17:00:58.329333 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.329283    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 17:00:58.329594 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.329442    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 17:00:58.329704 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.329684    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 17:00:58.330421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.330181    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 17:00:58.330421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.330194    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 17:00:58.331095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.331080    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6kcft"
Jul 27 17:00:58.333206 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.333184    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 17:00:58.333627 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.333608    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 17:00:58.333785 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.333764    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:00:58.333869 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.333783    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pbnxf\""
Jul 27 17:00:58.333924 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:58.333857    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8jgtw" podUID="16bb5aa2-183e-4742-a424-4ba30c7e3d95"
Jul 27 17:00:58.334893 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.334871    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-modprobe-d\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.335058 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.334909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/676386a3-b937-45a3-88e9-54649e0839e0-serviceca\") pod \"node-ca-2hn7j\" (UID: \"676386a3-b937-45a3-88e9-54649e0839e0\") " pod="openshift-image-registry/node-ca-2hn7j"
Jul 27 17:00:58.335058 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.334935    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kjvm\" (UniqueName: \"kubernetes.io/projected/676386a3-b937-45a3-88e9-54649e0839e0-kube-api-access-2kjvm\") pod \"node-ca-2hn7j\" (UID: \"676386a3-b937-45a3-88e9-54649e0839e0\") " pod="openshift-image-registry/node-ca-2hn7j"
Jul 27 17:00:58.335058 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.334960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edb57dae-199b-48e9-9a8c-592f3101d307-system-cni-dir\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.335058 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.334981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edb57dae-199b-48e9-9a8c-592f3101d307-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.335058 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335004    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/edb57dae-199b-48e9-9a8c-592f3101d307-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.335058 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335029    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44c9f115-51d1-4d4a-9a71-0c925017f191-etc-selinux\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.335395 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335074    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-sys\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.335395 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335106    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d-agent-certs\") pod \"konnectivity-agent-vclxn\" (UID: \"2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d\") " pod="kube-system/konnectivity-agent-vclxn"
Jul 27 17:00:58.335395 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335131    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44c9f115-51d1-4d4a-9a71-0c925017f191-device-dir\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.335395 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335187    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2phlx\" (UniqueName: \"kubernetes.io/projected/44c9f115-51d1-4d4a-9a71-0c925017f191-kube-api-access-2phlx\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.335395 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335245    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-sysctl-d\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.335395 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335274    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-systemd\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.335395 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-lib-modules\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.335395 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335336    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-var-lib-kubelet\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.335395 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335357    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-tuned\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.335395 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5htgl\" (UniqueName: \"kubernetes.io/projected/edb57dae-199b-48e9-9a8c-592f3101d307-kube-api-access-5htgl\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.335832 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335421    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44c9f115-51d1-4d4a-9a71-0c925017f191-registration-dir\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.335832 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335445    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-kubernetes\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.335832 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335477    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edb57dae-199b-48e9-9a8c-592f3101d307-cnibin\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.335832 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335503    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/edb57dae-199b-48e9-9a8c-592f3101d307-cni-binary-copy\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.335832 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335525    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-sysconfig\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.335832 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335551    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-run\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.335832 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335572    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10b7f9b7-2929-40bd-8e34-dad3d098187e-tmp\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.335832 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335593    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc657\" (UniqueName: \"kubernetes.io/projected/10b7f9b7-2929-40bd-8e34-dad3d098187e-kube-api-access-gc657\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.335832 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335628    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/edb57dae-199b-48e9-9a8c-592f3101d307-os-release\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.335832 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d-konnectivity-ca\") pod \"konnectivity-agent-vclxn\" (UID: \"2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d\") " pod="kube-system/konnectivity-agent-vclxn"
Jul 27 17:00:58.335832 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335674    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44c9f115-51d1-4d4a-9a71-0c925017f191-socket-dir\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.335832 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335708    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44c9f115-51d1-4d4a-9a71-0c925017f191-sys-fs\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.335832 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335734    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtc76\" (UniqueName: \"kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76\") pod \"network-check-target-6pt78\" (UID: \"0ab2a8bd-0d12-41c0-ae51-b4e3b262a896\") " pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:00:58.335832 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335788    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/edb57dae-199b-48e9-9a8c-592f3101d307-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.335832 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335830    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-sysctl-conf\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.336527 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335853    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-host\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.336527 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335873    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/676386a3-b937-45a3-88e9-54649e0839e0-host\") pod \"node-ca-2hn7j\" (UID: \"676386a3-b937-45a3-88e9-54649e0839e0\") " pod="openshift-image-registry/node-ca-2hn7j"
Jul 27 17:00:58.336527 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335894    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44c9f115-51d1-4d4a-9a71-0c925017f191-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.336527 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.335960    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 17:00:58.336527 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.336210    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9sh7d"
Jul 27 17:00:58.338308 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.338286    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:00:58.338470 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:58.338350    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d72jm" podUID="e0651122-b4fd-4a56-8909-53661d776bfc"
Jul 27 17:00:58.338470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.338376    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:00:58.338594 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.338563    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 17:00:58.338594 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.338580    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sz2bz\""
Jul 27 17:00:58.338795 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.338777    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 17:00:58.340653 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.340637    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.342851 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.342825    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nw7k9\""
Jul 27 17:00:58.342966 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.342933    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 17:00:58.360793 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.360774    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kqvvc"
Jul 27 17:00:58.367024 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.367009    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kqvvc"
Jul 27 17:00:58.404192 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.404138    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal" event={"ID":"81d1b794894ec66f164d640b4bfc3b9a","Type":"ContainerStarted","Data":"0e9f5b5050cd766cacac78522d5fd1b1a45c1202db639e210eab00812500889e"}
Jul 27 17:00:58.405292 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.405255    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal" event={"ID":"02541bc35c7e12918e006d46c38418ad","Type":"ContainerStarted","Data":"4f16efa4e40e1ad4e95637ecdbf7e70a0ab4a8ac57f22094358b83cf97826378"}
Jul 27 17:00:58.425208 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.425178    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 17:00:58.436135 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436107    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b606708-d7b9-4eab-8da8-ecfd50f006fb-ovnkube-config\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.436266 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e02881b5-7256-40d5-9ad4-199b7c349a48-tmp-dir\") pod \"node-resolver-6kcft\" (UID: \"e02881b5-7256-40d5-9ad4-199b7c349a48\") " pod="openshift-dns/node-resolver-6kcft"
Jul 27 17:00:58.436266 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d-konnectivity-ca\") pod \"konnectivity-agent-vclxn\" (UID: \"2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d\") " pod="kube-system/konnectivity-agent-vclxn"
Jul 27 17:00:58.436266 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436226    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44c9f115-51d1-4d4a-9a71-0c925017f191-sys-fs\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.436421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436275    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-systemd-units\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.436421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436308    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-etc-kubernetes\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.436421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436338    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtc76\" (UniqueName: \"kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76\") pod \"network-check-target-6pt78\" (UID: \"0ab2a8bd-0d12-41c0-ae51-b4e3b262a896\") " pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:00:58.436540 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436443    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44c9f115-51d1-4d4a-9a71-0c925017f191-sys-fs\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.436540 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436486    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-cni-bin\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.436540 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436533    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-system-cni-dir\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.436682 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436584    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-host-var-lib-kubelet\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.436682 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436628    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/edb57dae-199b-48e9-9a8c-592f3101d307-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.436682 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436658    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-host\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.436822 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436685    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.436822 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436710    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-cni-netd\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.436822 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-host-var-lib-cni-multus\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.436822 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436741    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-host\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.436822 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436758    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44c9f115-51d1-4d4a-9a71-0c925017f191-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.436822 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436812    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-multus-cni-dir\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.437132 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436838    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44c9f115-51d1-4d4a-9a71-0c925017f191-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.437132 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d-konnectivity-ca\") pod \"konnectivity-agent-vclxn\" (UID: \"2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d\") " pod="kube-system/konnectivity-agent-vclxn"
Jul 27 17:00:58.437132 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436843    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/676386a3-b937-45a3-88e9-54649e0839e0-serviceca\") pod \"node-ca-2hn7j\" (UID: \"676386a3-b937-45a3-88e9-54649e0839e0\") " pod="openshift-image-registry/node-ca-2hn7j"
Jul 27 17:00:58.437132 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436933    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kjvm\" (UniqueName: \"kubernetes.io/projected/676386a3-b937-45a3-88e9-54649e0839e0-kube-api-access-2kjvm\") pod \"node-ca-2hn7j\" (UID: \"676386a3-b937-45a3-88e9-54649e0839e0\") " pod="openshift-image-registry/node-ca-2hn7j"
Jul 27 17:00:58.437132 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436962    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edb57dae-199b-48e9-9a8c-592f3101d307-system-cni-dir\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.437132 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.436998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edb57dae-199b-48e9-9a8c-592f3101d307-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.437132 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437024    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44c9f115-51d1-4d4a-9a71-0c925017f191-etc-selinux\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.437132 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-sys\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.437132 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437073    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-lib-modules\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.437132 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437083    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/edb57dae-199b-48e9-9a8c-592f3101d307-system-cni-dir\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.437132 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437098    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2phlx\" (UniqueName: \"kubernetes.io/projected/44c9f115-51d1-4d4a-9a71-0c925017f191-kube-api-access-2phlx\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.437661 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437141    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/edb57dae-199b-48e9-9a8c-592f3101d307-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.437661 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437144    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-sysctl-d\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.437661 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437156    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44c9f115-51d1-4d4a-9a71-0c925017f191-etc-selinux\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.437661 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-var-lib-kubelet\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.437661 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437221    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-tuned\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.437661 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-slash\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.437661 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437269    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-log-socket\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.437661 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437291    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-sysctl-d\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.437661 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437293    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-run-openvswitch\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.437661 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437317    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/edb57dae-199b-48e9-9a8c-592f3101d307-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.437661 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437343    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-var-lib-kubelet\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.437661 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437375    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/676386a3-b937-45a3-88e9-54649e0839e0-serviceca\") pod \"node-ca-2hn7j\" (UID: \"676386a3-b937-45a3-88e9-54649e0839e0\") " pod="openshift-image-registry/node-ca-2hn7j"
Jul 27 17:00:58.437661 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs\") pod \"network-metrics-daemon-d72jm\" (UID: \"e0651122-b4fd-4a56-8909-53661d776bfc\") " pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:00:58.437661 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437420    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-sys\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.437661 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437443    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5htgl\" (UniqueName: \"kubernetes.io/projected/edb57dae-199b-48e9-9a8c-592f3101d307-kube-api-access-5htgl\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.437661 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-kubernetes\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.437661 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437522    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-lib-modules\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.438470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437572    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 17:00:58.438470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437600    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/16bb5aa2-183e-4742-a424-4ba30c7e3d95-dbus\") pod \"global-pull-secret-syncer-8jgtw\" (UID: \"16bb5aa2-183e-4742-a424-4ba30c7e3d95\") " pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:00:58.438470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437628    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-kubernetes\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.438470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437670    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdl72\" (UniqueName: \"kubernetes.io/projected/e0651122-b4fd-4a56-8909-53661d776bfc-kube-api-access-mdl72\") pod \"network-metrics-daemon-d72jm\" (UID: \"e0651122-b4fd-4a56-8909-53661d776bfc\") " pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:00:58.438470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437702    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-multus-socket-dir-parent\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.438470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437735    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-host-run-k8s-cni-cncf-io\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.438470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437758    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-host-run-netns\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.438470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437781    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-host-var-lib-cni-bin\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.438470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437824    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/edb57dae-199b-48e9-9a8c-592f3101d307-cni-binary-copy\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.438470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437859    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-sysconfig\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.438470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-run\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.438470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437908    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kx5b\" (UniqueName: \"kubernetes.io/projected/1b606708-d7b9-4eab-8da8-ecfd50f006fb-kube-api-access-7kx5b\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.438470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437931    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-cnibin\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.438470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437945    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-sysconfig\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.438470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-cni-binary-copy\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.438470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.437987    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-hostroot\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.438470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438008    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/edb57dae-199b-48e9-9a8c-592f3101d307-os-release\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.439278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438027    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44c9f115-51d1-4d4a-9a71-0c925017f191-socket-dir\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.439278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438052    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qjs5\" (UniqueName: \"kubernetes.io/projected/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-kube-api-access-2qjs5\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.439278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438062    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-run\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.439278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438078    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-etc-openvswitch\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.439278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438099    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-run-ovn\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.439278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438124    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/edb57dae-199b-48e9-9a8c-592f3101d307-os-release\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.439278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.439278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438159    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/16bb5aa2-183e-4742-a424-4ba30c7e3d95-kubelet-config\") pod \"global-pull-secret-syncer-8jgtw\" (UID: \"16bb5aa2-183e-4742-a424-4ba30c7e3d95\") " pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:00:58.439278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438214    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-sysctl-conf\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.439278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438239    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b606708-d7b9-4eab-8da8-ecfd50f006fb-env-overrides\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.439278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438263    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b606708-d7b9-4eab-8da8-ecfd50f006fb-ovnkube-script-lib\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.439278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44c9f115-51d1-4d4a-9a71-0c925017f191-socket-dir\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.439278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438287    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqtdj\" (UniqueName: \"kubernetes.io/projected/e02881b5-7256-40d5-9ad4-199b7c349a48-kube-api-access-sqtdj\") pod \"node-resolver-6kcft\" (UID: \"e02881b5-7256-40d5-9ad4-199b7c349a48\") " pod="openshift-dns/node-resolver-6kcft"
Jul 27 17:00:58.439278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438331    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-os-release\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.439278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438378    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/676386a3-b937-45a3-88e9-54649e0839e0-host\") pod \"node-ca-2hn7j\" (UID: \"676386a3-b937-45a3-88e9-54649e0839e0\") " pod="openshift-image-registry/node-ca-2hn7j"
Jul 27 17:00:58.439278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438377    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/edb57dae-199b-48e9-9a8c-592f3101d307-cni-binary-copy\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.439278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438418    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-modprobe-d\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.440043 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438442    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/676386a3-b937-45a3-88e9-54649e0839e0-host\") pod \"node-ca-2hn7j\" (UID: \"676386a3-b937-45a3-88e9-54649e0839e0\") " pod="openshift-image-registry/node-ca-2hn7j"
Jul 27 17:00:58.440043 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438452    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-sysctl-conf\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.440043 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438485    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-run-systemd\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.440043 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438516    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e02881b5-7256-40d5-9ad4-199b7c349a48-hosts-file\") pod \"node-resolver-6kcft\" (UID: \"e02881b5-7256-40d5-9ad4-199b7c349a48\") " pod="openshift-dns/node-resolver-6kcft"
Jul 27 17:00:58.440043 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438541    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-modprobe-d\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.440043 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438548    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acad7c3c-4a24-45a5-b9e3-9dbdd6319d02-iptables-alerter-script\") pod \"iptables-alerter-9sh7d\" (UID: \"acad7c3c-4a24-45a5-b9e3-9dbdd6319d02\") " pod="openshift-network-operator/iptables-alerter-9sh7d"
Jul 27 17:00:58.440043 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vw46\" (UniqueName: \"kubernetes.io/projected/acad7c3c-4a24-45a5-b9e3-9dbdd6319d02-kube-api-access-8vw46\") pod \"iptables-alerter-9sh7d\" (UID: \"acad7c3c-4a24-45a5-b9e3-9dbdd6319d02\") " pod="openshift-network-operator/iptables-alerter-9sh7d"
Jul 27 17:00:58.440043 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438607    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-multus-daemon-config\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.440043 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438636    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/edb57dae-199b-48e9-9a8c-592f3101d307-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.440043 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438677    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10b7f9b7-2929-40bd-8e34-dad3d098187e-tmp\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.440043 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438701    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc657\" (UniqueName: \"kubernetes.io/projected/10b7f9b7-2929-40bd-8e34-dad3d098187e-kube-api-access-gc657\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.440043 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438726    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-kubelet\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.440043 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438749    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d-agent-certs\") pod \"konnectivity-agent-vclxn\" (UID: \"2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d\") " pod="kube-system/konnectivity-agent-vclxn"
Jul 27 17:00:58.440043 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44c9f115-51d1-4d4a-9a71-0c925017f191-device-dir\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.440043 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438794    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-systemd\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.440043 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438816    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-node-log\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.440043 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438839    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret\") pod \"global-pull-secret-syncer-8jgtw\" (UID: \"16bb5aa2-183e-4742-a424-4ba30c7e3d95\") " pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:00:58.440823 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438861    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acad7c3c-4a24-45a5-b9e3-9dbdd6319d02-host-slash\") pod \"iptables-alerter-9sh7d\" (UID: \"acad7c3c-4a24-45a5-b9e3-9dbdd6319d02\") " pod="openshift-network-operator/iptables-alerter-9sh7d"
Jul 27 17:00:58.440823 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438884    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-host-run-multus-certs\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.440823 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438915    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44c9f115-51d1-4d4a-9a71-0c925017f191-registration-dir\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.440823 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438940    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b606708-d7b9-4eab-8da8-ecfd50f006fb-ovn-node-metrics-cert\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.440823 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438962    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-multus-conf-dir\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.440823 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.438987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edb57dae-199b-48e9-9a8c-592f3101d307-cnibin\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.440823 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.439037    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-run-netns\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.440823 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.439063    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44c9f115-51d1-4d4a-9a71-0c925017f191-device-dir\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.440823 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.439081    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-var-lib-openvswitch\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.440823 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.439123    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-systemd\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.440823 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.439153    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/edb57dae-199b-48e9-9a8c-592f3101d307-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.440823 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.439241    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44c9f115-51d1-4d4a-9a71-0c925017f191-registration-dir\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.440823 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.439352    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/edb57dae-199b-48e9-9a8c-592f3101d307-cnibin\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.442129 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:58.441810    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:00:58.442129 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:58.441835    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:00:58.442129 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:58.441859    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gtc76 for pod openshift-network-diagnostics/network-check-target-6pt78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:00:58.442129 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:58.441928    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76 podName:0ab2a8bd-0d12-41c0-ae51-b4e3b262a896 nodeName:}" failed. No retries permitted until 2026-07-27 17:00:58.941901767 +0000 UTC m=+3.017166235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gtc76" (UniqueName: "kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76") pod "network-check-target-6pt78" (UID: "0ab2a8bd-0d12-41c0-ae51-b4e3b262a896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:00:58.443199 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.442879    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/10b7f9b7-2929-40bd-8e34-dad3d098187e-etc-tuned\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.443199 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.443094    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10b7f9b7-2929-40bd-8e34-dad3d098187e-tmp\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.443455 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.443435    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d-agent-certs\") pod \"konnectivity-agent-vclxn\" (UID: \"2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d\") " pod="kube-system/konnectivity-agent-vclxn"
Jul 27 17:00:58.445758 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.445662    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5htgl\" (UniqueName: \"kubernetes.io/projected/edb57dae-199b-48e9-9a8c-592f3101d307-kube-api-access-5htgl\") pod \"multus-additional-cni-plugins-2mjl8\" (UID: \"edb57dae-199b-48e9-9a8c-592f3101d307\") " pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.445758 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.445716    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kjvm\" (UniqueName: \"kubernetes.io/projected/676386a3-b937-45a3-88e9-54649e0839e0-kube-api-access-2kjvm\") pod \"node-ca-2hn7j\" (UID: \"676386a3-b937-45a3-88e9-54649e0839e0\") " pod="openshift-image-registry/node-ca-2hn7j"
Jul 27 17:00:58.446547 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.446524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2phlx\" (UniqueName: \"kubernetes.io/projected/44c9f115-51d1-4d4a-9a71-0c925017f191-kube-api-access-2phlx\") pod \"aws-ebs-csi-driver-node-wp9wk\" (UID: \"44c9f115-51d1-4d4a-9a71-0c925017f191\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.447290 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.447269    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc657\" (UniqueName: \"kubernetes.io/projected/10b7f9b7-2929-40bd-8e34-dad3d098187e-kube-api-access-gc657\") pod \"tuned-vh2nd\" (UID: \"10b7f9b7-2929-40bd-8e34-dad3d098187e\") " pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.539952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.539925    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e02881b5-7256-40d5-9ad4-199b7c349a48-tmp-dir\") pod \"node-resolver-6kcft\" (UID: \"e02881b5-7256-40d5-9ad4-199b7c349a48\") " pod="openshift-dns/node-resolver-6kcft"
Jul 27 17:00:58.539952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.539956    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-systemd-units\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.540191 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.539974    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-etc-kubernetes\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.540191 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540046    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-etc-kubernetes\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.540191 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540054    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-systemd-units\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.540191 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540089    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-cni-bin\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.540191 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-system-cni-dir\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.540191 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540138    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-host-var-lib-kubelet\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.540421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540206    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.540421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540232    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-cni-netd\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.540421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540250    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-cni-bin\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.540421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540266    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.540421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540229    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-host-var-lib-kubelet\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.540421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540250    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-system-cni-dir\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.540421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-host-var-lib-cni-multus\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.540421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-cni-netd\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.540421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540303    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e02881b5-7256-40d5-9ad4-199b7c349a48-tmp-dir\") pod \"node-resolver-6kcft\" (UID: \"e02881b5-7256-40d5-9ad4-199b7c349a48\") " pod="openshift-dns/node-resolver-6kcft"
Jul 27 17:00:58.540421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540307    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-host-var-lib-cni-multus\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.540421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-multus-cni-dir\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.540421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540401    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-slash\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.540421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540427    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-log-socket\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.540952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540435    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-multus-cni-dir\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.540952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540450    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-run-openvswitch\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.540952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540464    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-slash\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.540952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540473    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs\") pod \"network-metrics-daemon-d72jm\" (UID: \"e0651122-b4fd-4a56-8909-53661d776bfc\") " pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:00:58.540952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540499    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/16bb5aa2-183e-4742-a424-4ba30c7e3d95-dbus\") pod \"global-pull-secret-syncer-8jgtw\" (UID: \"16bb5aa2-183e-4742-a424-4ba30c7e3d95\") " pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:00:58.540952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540503    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-run-openvswitch\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.540952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540539    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdl72\" (UniqueName: \"kubernetes.io/projected/e0651122-b4fd-4a56-8909-53661d776bfc-kube-api-access-mdl72\") pod \"network-metrics-daemon-d72jm\" (UID: \"e0651122-b4fd-4a56-8909-53661d776bfc\") " pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:00:58.540952 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:58.540560    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:00:58.540952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-multus-socket-dir-parent\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.540952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540593    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-host-run-k8s-cni-cncf-io\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.540952 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:58.540619    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs podName:e0651122-b4fd-4a56-8909-53661d776bfc nodeName:}" failed. No retries permitted until 2026-07-27 17:00:59.04060057 +0000 UTC m=+3.115865043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs") pod "network-metrics-daemon-d72jm" (UID: "e0651122-b4fd-4a56-8909-53661d776bfc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:00:58.540952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540624    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-host-run-k8s-cni-cncf-io\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.540952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540648    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-host-run-netns\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.540952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540672    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-multus-socket-dir-parent\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.540952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540679    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-host-var-lib-cni-bin\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.540952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540709    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-host-run-netns\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.540952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540709    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kx5b\" (UniqueName: \"kubernetes.io/projected/1b606708-d7b9-4eab-8da8-ecfd50f006fb-kube-api-access-7kx5b\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.541711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540742    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-cnibin\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.541711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540764    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-cni-binary-copy\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.541711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-hostroot\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.541711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540476    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-log-socket\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.541711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540847    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-cnibin\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.541711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540880    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-host-var-lib-cni-bin\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.541711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540888    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-hostroot\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.541711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540912    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qjs5\" (UniqueName: \"kubernetes.io/projected/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-kube-api-access-2qjs5\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.541711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-etc-openvswitch\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.541711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540978    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-run-ovn\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.541711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540989    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/16bb5aa2-183e-4742-a424-4ba30c7e3d95-dbus\") pod \"global-pull-secret-syncer-8jgtw\" (UID: \"16bb5aa2-183e-4742-a424-4ba30c7e3d95\") " pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:00:58.541711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.540997    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-etc-openvswitch\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.541711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541005    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.541711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541430    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/16bb5aa2-183e-4742-a424-4ba30c7e3d95-kubelet-config\") pod \"global-pull-secret-syncer-8jgtw\" (UID: \"16bb5aa2-183e-4742-a424-4ba30c7e3d95\") " pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:00:58.541711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541469    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b606708-d7b9-4eab-8da8-ecfd50f006fb-env-overrides\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.541711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541511    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b606708-d7b9-4eab-8da8-ecfd50f006fb-ovnkube-script-lib\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.541711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541543    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqtdj\" (UniqueName: \"kubernetes.io/projected/e02881b5-7256-40d5-9ad4-199b7c349a48-kube-api-access-sqtdj\") pod \"node-resolver-6kcft\" (UID: \"e02881b5-7256-40d5-9ad4-199b7c349a48\") " pod="openshift-dns/node-resolver-6kcft"
Jul 27 17:00:58.541711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541592    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-os-release\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.542592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541612    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/16bb5aa2-183e-4742-a424-4ba30c7e3d95-kubelet-config\") pod \"global-pull-secret-syncer-8jgtw\" (UID: \"16bb5aa2-183e-4742-a424-4ba30c7e3d95\") " pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:00:58.542592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541618    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-run-ovn\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.542592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541627    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-run-systemd\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.542592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541613    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-cni-binary-copy\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.542592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541679    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e02881b5-7256-40d5-9ad4-199b7c349a48-hosts-file\") pod \"node-resolver-6kcft\" (UID: \"e02881b5-7256-40d5-9ad4-199b7c349a48\") " pod="openshift-dns/node-resolver-6kcft"
Jul 27 17:00:58.542592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541678    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.542592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-os-release\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.542592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541729    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acad7c3c-4a24-45a5-b9e3-9dbdd6319d02-iptables-alerter-script\") pod \"iptables-alerter-9sh7d\" (UID: \"acad7c3c-4a24-45a5-b9e3-9dbdd6319d02\") " pod="openshift-network-operator/iptables-alerter-9sh7d"
Jul 27 17:00:58.542592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541755    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e02881b5-7256-40d5-9ad4-199b7c349a48-hosts-file\") pod \"node-resolver-6kcft\" (UID: \"e02881b5-7256-40d5-9ad4-199b7c349a48\") " pod="openshift-dns/node-resolver-6kcft"
Jul 27 17:00:58.542592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541700    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-run-systemd\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.542592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541770    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vw46\" (UniqueName: \"kubernetes.io/projected/acad7c3c-4a24-45a5-b9e3-9dbdd6319d02-kube-api-access-8vw46\") pod \"iptables-alerter-9sh7d\" (UID: \"acad7c3c-4a24-45a5-b9e3-9dbdd6319d02\") " pod="openshift-network-operator/iptables-alerter-9sh7d"
Jul 27 17:00:58.542592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541825    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-multus-daemon-config\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.542592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541875    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-kubelet\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.542592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-node-log\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.542592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541942    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret\") pod \"global-pull-secret-syncer-8jgtw\" (UID: \"16bb5aa2-183e-4742-a424-4ba30c7e3d95\") " pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:00:58.542592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.541969    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acad7c3c-4a24-45a5-b9e3-9dbdd6319d02-host-slash\") pod \"iptables-alerter-9sh7d\" (UID: \"acad7c3c-4a24-45a5-b9e3-9dbdd6319d02\") " pod="openshift-network-operator/iptables-alerter-9sh7d"
Jul 27 17:00:58.542592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.542000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-host-run-multus-certs\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.543378 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.542034    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b606708-d7b9-4eab-8da8-ecfd50f006fb-ovn-node-metrics-cert\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.543378 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.542068    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-multus-conf-dir\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.543378 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.542075    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b606708-d7b9-4eab-8da8-ecfd50f006fb-env-overrides\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.543378 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.542099    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-run-netns\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.543378 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.542112    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acad7c3c-4a24-45a5-b9e3-9dbdd6319d02-host-slash\") pod \"iptables-alerter-9sh7d\" (UID: \"acad7c3c-4a24-45a5-b9e3-9dbdd6319d02\") " pod="openshift-network-operator/iptables-alerter-9sh7d"
Jul 27 17:00:58.543378 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.542126    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-var-lib-openvswitch\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.543378 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.542157    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b606708-d7b9-4eab-8da8-ecfd50f006fb-ovnkube-config\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.543378 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:58.542233    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:00:58.543378 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:58.542289    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret podName:16bb5aa2-183e-4742-a424-4ba30c7e3d95 nodeName:}" failed. No retries permitted until 2026-07-27 17:00:59.042272704 +0000 UTC m=+3.117537171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret") pod "global-pull-secret-syncer-8jgtw" (UID: "16bb5aa2-183e-4742-a424-4ba30c7e3d95") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:00:58.543378 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.542481    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-kubelet\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.543378 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.542502    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-host-run-netns\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.543378 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.542077    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-node-log\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.543378 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.542571    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-host-run-multus-certs\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.543378 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.542582    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b606708-d7b9-4eab-8da8-ecfd50f006fb-var-lib-openvswitch\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.543378 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.542597    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-multus-conf-dir\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.543378 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.542786    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acad7c3c-4a24-45a5-b9e3-9dbdd6319d02-iptables-alerter-script\") pod \"iptables-alerter-9sh7d\" (UID: \"acad7c3c-4a24-45a5-b9e3-9dbdd6319d02\") " pod="openshift-network-operator/iptables-alerter-9sh7d"
Jul 27 17:00:58.544040 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.543416    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b606708-d7b9-4eab-8da8-ecfd50f006fb-ovnkube-script-lib\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.544583 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.544548    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-multus-daemon-config\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.545598 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.545561    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b606708-d7b9-4eab-8da8-ecfd50f006fb-ovnkube-config\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.546635 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.546612    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b606708-d7b9-4eab-8da8-ecfd50f006fb-ovn-node-metrics-cert\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.554099 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.554073    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdl72\" (UniqueName: \"kubernetes.io/projected/e0651122-b4fd-4a56-8909-53661d776bfc-kube-api-access-mdl72\") pod \"network-metrics-daemon-d72jm\" (UID: \"e0651122-b4fd-4a56-8909-53661d776bfc\") " pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:00:58.554591 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.554567    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kx5b\" (UniqueName: \"kubernetes.io/projected/1b606708-d7b9-4eab-8da8-ecfd50f006fb-kube-api-access-7kx5b\") pod \"ovnkube-node-m9k9v\" (UID: \"1b606708-d7b9-4eab-8da8-ecfd50f006fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.554687 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.554593    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqtdj\" (UniqueName: \"kubernetes.io/projected/e02881b5-7256-40d5-9ad4-199b7c349a48-kube-api-access-sqtdj\") pod \"node-resolver-6kcft\" (UID: \"e02881b5-7256-40d5-9ad4-199b7c349a48\") " pod="openshift-dns/node-resolver-6kcft"
Jul 27 17:00:58.555287 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.555247    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qjs5\" (UniqueName: \"kubernetes.io/projected/952b4a9c-f2f9-4bf9-a02b-cc426d1995c1-kube-api-access-2qjs5\") pod \"multus-8m5b9\" (UID: \"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1\") " pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.555736 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.555710    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vw46\" (UniqueName: \"kubernetes.io/projected/acad7c3c-4a24-45a5-b9e3-9dbdd6319d02-kube-api-access-8vw46\") pod \"iptables-alerter-9sh7d\" (UID: \"acad7c3c-4a24-45a5-b9e3-9dbdd6319d02\") " pod="openshift-network-operator/iptables-alerter-9sh7d"
Jul 27 17:00:58.624654 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.624573    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2hn7j"
Jul 27 17:00:58.633796 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.633774    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2mjl8"
Jul 27 17:00:58.642575 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.642551    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vclxn"
Jul 27 17:00:58.648144 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.648126    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk"
Jul 27 17:00:58.656813 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.656794    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vh2nd"
Jul 27 17:00:58.664533 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.664517    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:00:58.671119 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.671104    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6kcft"
Jul 27 17:00:58.679644 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.679624    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9sh7d"
Jul 27 17:00:58.686226 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.686209    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8m5b9"
Jul 27 17:00:58.944125 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:58.944094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtc76\" (UniqueName: \"kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76\") pod \"network-check-target-6pt78\" (UID: \"0ab2a8bd-0d12-41c0-ae51-b4e3b262a896\") " pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:00:58.944308 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:58.944288    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:00:58.944383 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:58.944314    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:00:58.944383 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:58.944327    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gtc76 for pod openshift-network-diagnostics/network-check-target-6pt78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:00:58.944504 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:58.944389    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76 podName:0ab2a8bd-0d12-41c0-ae51-b4e3b262a896 nodeName:}" failed. No retries permitted until 2026-07-27 17:00:59.944369499 +0000 UTC m=+4.019633968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gtc76" (UniqueName: "kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76") pod "network-check-target-6pt78" (UID: "0ab2a8bd-0d12-41c0-ae51-b4e3b262a896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:00:59.045360 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:59.045333    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs\") pod \"network-metrics-daemon-d72jm\" (UID: \"e0651122-b4fd-4a56-8909-53661d776bfc\") " pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:00:59.045529 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:59.045375    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret\") pod \"global-pull-secret-syncer-8jgtw\" (UID: \"16bb5aa2-183e-4742-a424-4ba30c7e3d95\") " pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:00:59.045529 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:59.045469    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:00:59.045529 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:59.045487    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:00:59.045529 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:59.045514    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret podName:16bb5aa2-183e-4742-a424-4ba30c7e3d95 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:00.045502092 +0000 UTC m=+4.120766557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret") pod "global-pull-secret-syncer-8jgtw" (UID: "16bb5aa2-183e-4742-a424-4ba30c7e3d95") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:00:59.045691 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:59.045547    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs podName:e0651122-b4fd-4a56-8909-53661d776bfc nodeName:}" failed. No retries permitted until 2026-07-27 17:01:00.045529704 +0000 UTC m=+4.120794169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs") pod "network-metrics-daemon-d72jm" (UID: "e0651122-b4fd-4a56-8909-53661d776bfc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:00:59.268774 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:59.268698    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:00:59.370246 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:59.370184    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 16:55:58 +0000 UTC" deadline="2028-01-14 03:45:13.422742504 +0000 UTC"
Jul 27 17:00:59.370246 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:59.370240    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12850h44m14.052506655s"
Jul 27 17:00:59.425082 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:59.424994    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02881b5_7256_40d5_9ad4_199b7c349a48.slice/crio-34a04b6a5a6f94f8a5b3145175dbf69728ca362781f2fadc7fe2ee005ecdff13 WatchSource:0}: Error finding container 34a04b6a5a6f94f8a5b3145175dbf69728ca362781f2fadc7fe2ee005ecdff13: Status 404 returned error can't find the container with id 34a04b6a5a6f94f8a5b3145175dbf69728ca362781f2fadc7fe2ee005ecdff13
Jul 27 17:00:59.426424 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:59.426396    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5e96ae_f5a4_4e64_9c29_1dbcb1f9cc4d.slice/crio-2c00fb808a5f931c1e93fffc797704cba910cba3d2703f5da66547c57fca33db WatchSource:0}: Error finding container 2c00fb808a5f931c1e93fffc797704cba910cba3d2703f5da66547c57fca33db: Status 404 returned error can't find the container with id 2c00fb808a5f931c1e93fffc797704cba910cba3d2703f5da66547c57fca33db
Jul 27 17:00:59.429365 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:59.429339    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b606708_d7b9_4eab_8da8_ecfd50f006fb.slice/crio-5235b58652224c5ece8edde7e9735e2a714c8fc85b44fdfb1a7207548d80ad18 WatchSource:0}: Error finding container 5235b58652224c5ece8edde7e9735e2a714c8fc85b44fdfb1a7207548d80ad18: Status 404 returned error can't find the container with id 5235b58652224c5ece8edde7e9735e2a714c8fc85b44fdfb1a7207548d80ad18
Jul 27 17:00:59.431277 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:59.431244    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676386a3_b937_45a3_88e9_54649e0839e0.slice/crio-77db6ab77713647b5938c6f72b12cb51ea358dd02bd0d7b9b022812036b2cb06 WatchSource:0}: Error finding container 77db6ab77713647b5938c6f72b12cb51ea358dd02bd0d7b9b022812036b2cb06: Status 404 returned error can't find the container with id 77db6ab77713647b5938c6f72b12cb51ea358dd02bd0d7b9b022812036b2cb06
Jul 27 17:00:59.432241 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:59.432219    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b7f9b7_2929_40bd_8e34_dad3d098187e.slice/crio-18d32c321423ade64e33868c4ff807830b260faed3a839ee5706b42da68aafea WatchSource:0}: Error finding container 18d32c321423ade64e33868c4ff807830b260faed3a839ee5706b42da68aafea: Status 404 returned error can't find the container with id 18d32c321423ade64e33868c4ff807830b260faed3a839ee5706b42da68aafea
Jul 27 17:00:59.433509 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:59.432967    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacad7c3c_4a24_45a5_b9e3_9dbdd6319d02.slice/crio-a10422172b76c92bfe16179b7a2a0404a19eef0e9eae4f601f74ce90a68384bb WatchSource:0}: Error finding container a10422172b76c92bfe16179b7a2a0404a19eef0e9eae4f601f74ce90a68384bb: Status 404 returned error can't find the container with id a10422172b76c92bfe16179b7a2a0404a19eef0e9eae4f601f74ce90a68384bb
Jul 27 17:00:59.434047 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:59.434007    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c9f115_51d1_4d4a_9a71_0c925017f191.slice/crio-9b69207974b7b390031e73e0369c98b7aecc0e5f70a23fef93fbfc64135966ca WatchSource:0}: Error finding container 9b69207974b7b390031e73e0369c98b7aecc0e5f70a23fef93fbfc64135966ca: Status 404 returned error can't find the container with id 9b69207974b7b390031e73e0369c98b7aecc0e5f70a23fef93fbfc64135966ca
Jul 27 17:00:59.435106 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:59.434886    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb57dae_199b_48e9_9a8c_592f3101d307.slice/crio-0acdb694ccfcbcdb554442b462a55fde4d5f474252a49a02619d583519d9f53e WatchSource:0}: Error finding container 0acdb694ccfcbcdb554442b462a55fde4d5f474252a49a02619d583519d9f53e: Status 404 returned error can't find the container with id 0acdb694ccfcbcdb554442b462a55fde4d5f474252a49a02619d583519d9f53e
Jul 27 17:00:59.436552 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:00:59.436529    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952b4a9c_f2f9_4bf9_a02b_cc426d1995c1.slice/crio-8f3b4c01f5c766e85f4eae04406df97b591cbfe7eab33c2fa973a5ddf0623f66 WatchSource:0}: Error finding container 8f3b4c01f5c766e85f4eae04406df97b591cbfe7eab33c2fa973a5ddf0623f66: Status 404 returned error can't find the container with id 8f3b4c01f5c766e85f4eae04406df97b591cbfe7eab33c2fa973a5ddf0623f66
Jul 27 17:00:59.952199 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:00:59.951933    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtc76\" (UniqueName: \"kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76\") pod \"network-check-target-6pt78\" (UID: \"0ab2a8bd-0d12-41c0-ae51-b4e3b262a896\") " pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:00:59.952376 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:59.952246    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:00:59.952376 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:59.952281    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:00:59.952376 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:59.952295    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gtc76 for pod openshift-network-diagnostics/network-check-target-6pt78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:00:59.952376 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:00:59.952361    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76 podName:0ab2a8bd-0d12-41c0-ae51-b4e3b262a896 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:01.952341131 +0000 UTC m=+6.027605614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gtc76" (UniqueName: "kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76") pod "network-check-target-6pt78" (UID: "0ab2a8bd-0d12-41c0-ae51-b4e3b262a896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:00.053600 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.053526    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret\") pod \"global-pull-secret-syncer-8jgtw\" (UID: \"16bb5aa2-183e-4742-a424-4ba30c7e3d95\") " pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:00.053771 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.053655    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs\") pod \"network-metrics-daemon-d72jm\" (UID: \"e0651122-b4fd-4a56-8909-53661d776bfc\") " pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:00.053843 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:00.053783    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:00.053893 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:00.053848    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs podName:e0651122-b4fd-4a56-8909-53661d776bfc nodeName:}" failed. No retries permitted until 2026-07-27 17:01:02.053823704 +0000 UTC m=+6.129088174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs") pod "network-metrics-daemon-d72jm" (UID: "e0651122-b4fd-4a56-8909-53661d776bfc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:00.054274 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:00.054253    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:00.054382 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:00.054317    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret podName:16bb5aa2-183e-4742-a424-4ba30c7e3d95 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:02.054302627 +0000 UTC m=+6.129567096 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret") pod "global-pull-secret-syncer-8jgtw" (UID: "16bb5aa2-183e-4742-a424-4ba30c7e3d95") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:00.370481 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.370333    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 16:55:58 +0000 UTC" deadline="2028-02-07 19:32:32.36415521 +0000 UTC"
Jul 27 17:01:00.370481 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.370371    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13442h31m31.993789238s"
Jul 27 17:01:00.399064 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.399024    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:00.399262 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:00.399158    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8jgtw" podUID="16bb5aa2-183e-4742-a424-4ba30c7e3d95"
Jul 27 17:01:00.399601 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.399375    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:00.399601 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.399462    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:00.399601 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:00.399558    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pt78" podUID="0ab2a8bd-0d12-41c0-ae51-b4e3b262a896"
Jul 27 17:01:00.399796 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:00.399661    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d72jm" podUID="e0651122-b4fd-4a56-8909-53661d776bfc"
Jul 27 17:01:00.413575 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.413540    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8m5b9" event={"ID":"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1","Type":"ContainerStarted","Data":"8f3b4c01f5c766e85f4eae04406df97b591cbfe7eab33c2fa973a5ddf0623f66"}
Jul 27 17:01:00.415766 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.415673    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk" event={"ID":"44c9f115-51d1-4d4a-9a71-0c925017f191","Type":"ContainerStarted","Data":"9b69207974b7b390031e73e0369c98b7aecc0e5f70a23fef93fbfc64135966ca"}
Jul 27 17:01:00.421311 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.421237    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9sh7d" event={"ID":"acad7c3c-4a24-45a5-b9e3-9dbdd6319d02","Type":"ContainerStarted","Data":"a10422172b76c92bfe16179b7a2a0404a19eef0e9eae4f601f74ce90a68384bb"}
Jul 27 17:01:00.427315 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.427258    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6kcft" event={"ID":"e02881b5-7256-40d5-9ad4-199b7c349a48","Type":"ContainerStarted","Data":"34a04b6a5a6f94f8a5b3145175dbf69728ca362781f2fadc7fe2ee005ecdff13"}
Jul 27 17:01:00.433892 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.433867    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal" event={"ID":"81d1b794894ec66f164d640b4bfc3b9a","Type":"ContainerStarted","Data":"9768a7cef2b13668c26523dfb5b5af9219a1f737672fda3c4faa71d7afe64cde"}
Jul 27 17:01:00.439579 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.439552    2566 generic.go:356] "Generic (PLEG): container finished" podID="02541bc35c7e12918e006d46c38418ad" containerID="68732e669ad32b0829af2a94e6fd53ea7b9684fa4c6b8eeb395b4f5a3ff54c5f" exitCode=0
Jul 27 17:01:00.439701 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.439631    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal" event={"ID":"02541bc35c7e12918e006d46c38418ad","Type":"ContainerDied","Data":"68732e669ad32b0829af2a94e6fd53ea7b9684fa4c6b8eeb395b4f5a3ff54c5f"}
Jul 27 17:01:00.449445 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.449323    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-86.ec2.internal" podStartSLOduration=3.449304209 podStartE2EDuration="3.449304209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:00.448018242 +0000 UTC m=+4.523282730" watchObservedRunningTime="2026-07-27 17:01:00.449304209 +0000 UTC m=+4.524568697"
Jul 27 17:01:00.451911 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.451886    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mjl8" event={"ID":"edb57dae-199b-48e9-9a8c-592f3101d307","Type":"ContainerStarted","Data":"0acdb694ccfcbcdb554442b462a55fde4d5f474252a49a02619d583519d9f53e"}
Jul 27 17:01:00.453832 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.453809    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vh2nd" event={"ID":"10b7f9b7-2929-40bd-8e34-dad3d098187e","Type":"ContainerStarted","Data":"18d32c321423ade64e33868c4ff807830b260faed3a839ee5706b42da68aafea"}
Jul 27 17:01:00.458685 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.458619    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2hn7j" event={"ID":"676386a3-b937-45a3-88e9-54649e0839e0","Type":"ContainerStarted","Data":"77db6ab77713647b5938c6f72b12cb51ea358dd02bd0d7b9b022812036b2cb06"}
Jul 27 17:01:00.463925 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.463817    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v" event={"ID":"1b606708-d7b9-4eab-8da8-ecfd50f006fb","Type":"ContainerStarted","Data":"5235b58652224c5ece8edde7e9735e2a714c8fc85b44fdfb1a7207548d80ad18"}
Jul 27 17:01:00.468491 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:00.468466    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vclxn" event={"ID":"2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d","Type":"ContainerStarted","Data":"2c00fb808a5f931c1e93fffc797704cba910cba3d2703f5da66547c57fca33db"}
Jul 27 17:01:01.490841 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:01.490801    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal" event={"ID":"02541bc35c7e12918e006d46c38418ad","Type":"ContainerStarted","Data":"4e0a36417c2c6514fc486d1ce1c631e4dd68d52747e972c3f849640d962f5fcb"}
Jul 27 17:01:01.969211 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:01.968542    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtc76\" (UniqueName: \"kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76\") pod \"network-check-target-6pt78\" (UID: \"0ab2a8bd-0d12-41c0-ae51-b4e3b262a896\") " pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:01.969211 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:01.968700    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:01:01.969211 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:01.968719    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:01:01.969211 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:01.968731    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gtc76 for pod openshift-network-diagnostics/network-check-target-6pt78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:01.969211 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:01.968818    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76 podName:0ab2a8bd-0d12-41c0-ae51-b4e3b262a896 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:05.968775542 +0000 UTC m=+10.044040030 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gtc76" (UniqueName: "kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76") pod "network-check-target-6pt78" (UID: "0ab2a8bd-0d12-41c0-ae51-b4e3b262a896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:02.068955 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:02.068916    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs\") pod \"network-metrics-daemon-d72jm\" (UID: \"e0651122-b4fd-4a56-8909-53661d776bfc\") " pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:02.069141 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:02.068988    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret\") pod \"global-pull-secret-syncer-8jgtw\" (UID: \"16bb5aa2-183e-4742-a424-4ba30c7e3d95\") " pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:02.069233 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:02.069151    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:02.069288 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:02.069241    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs podName:e0651122-b4fd-4a56-8909-53661d776bfc nodeName:}" failed. No retries permitted until 2026-07-27 17:01:06.06921896 +0000 UTC m=+10.144483430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs") pod "network-metrics-daemon-d72jm" (UID: "e0651122-b4fd-4a56-8909-53661d776bfc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:02.069696 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:02.069652    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:02.069804 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:02.069711    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret podName:16bb5aa2-183e-4742-a424-4ba30c7e3d95 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:06.069695342 +0000 UTC m=+10.144959813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret") pod "global-pull-secret-syncer-8jgtw" (UID: "16bb5aa2-183e-4742-a424-4ba30c7e3d95") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:02.399900 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:02.399820    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:02.400049 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:02.399948    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8jgtw" podUID="16bb5aa2-183e-4742-a424-4ba30c7e3d95"
Jul 27 17:01:02.400225 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:02.400178    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:02.400225 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:02.400195    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:02.400343 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:02.400280    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d72jm" podUID="e0651122-b4fd-4a56-8909-53661d776bfc"
Jul 27 17:01:02.400396 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:02.400347    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pt78" podUID="0ab2a8bd-0d12-41c0-ae51-b4e3b262a896"
Jul 27 17:01:04.399892 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:04.399861    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:04.400361 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:04.399989    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d72jm" podUID="e0651122-b4fd-4a56-8909-53661d776bfc"
Jul 27 17:01:04.400426 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:04.400407    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:04.400684 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:04.400502    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pt78" podUID="0ab2a8bd-0d12-41c0-ae51-b4e3b262a896"
Jul 27 17:01:04.400684 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:04.400562    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:04.400684 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:04.400638    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8jgtw" podUID="16bb5aa2-183e-4742-a424-4ba30c7e3d95"
Jul 27 17:01:06.001421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:06.001156    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtc76\" (UniqueName: \"kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76\") pod \"network-check-target-6pt78\" (UID: \"0ab2a8bd-0d12-41c0-ae51-b4e3b262a896\") " pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:06.001421 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:06.001366    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:01:06.001421 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:06.001386    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:01:06.001421 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:06.001398    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gtc76 for pod openshift-network-diagnostics/network-check-target-6pt78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:06.002053 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:06.001461    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76 podName:0ab2a8bd-0d12-41c0-ae51-b4e3b262a896 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:14.001444047 +0000 UTC m=+18.076708547 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gtc76" (UniqueName: "kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76") pod "network-check-target-6pt78" (UID: "0ab2a8bd-0d12-41c0-ae51-b4e3b262a896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:06.101929 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:06.101823    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret\") pod \"global-pull-secret-syncer-8jgtw\" (UID: \"16bb5aa2-183e-4742-a424-4ba30c7e3d95\") " pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:06.101929 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:06.101918    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs\") pod \"network-metrics-daemon-d72jm\" (UID: \"e0651122-b4fd-4a56-8909-53661d776bfc\") " pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:06.102217 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:06.101974    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:06.102217 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:06.102046    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret podName:16bb5aa2-183e-4742-a424-4ba30c7e3d95 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:14.102030201 +0000 UTC m=+18.177294682 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret") pod "global-pull-secret-syncer-8jgtw" (UID: "16bb5aa2-183e-4742-a424-4ba30c7e3d95") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:06.102217 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:06.102051    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:06.102217 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:06.102107    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs podName:e0651122-b4fd-4a56-8909-53661d776bfc nodeName:}" failed. No retries permitted until 2026-07-27 17:01:14.10208968 +0000 UTC m=+18.177354159 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs") pod "network-metrics-daemon-d72jm" (UID: "e0651122-b4fd-4a56-8909-53661d776bfc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:06.397708 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:06.397563    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:06.397708 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:06.397570    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:06.398739 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:06.398707    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8jgtw" podUID="16bb5aa2-183e-4742-a424-4ba30c7e3d95"
Jul 27 17:01:06.399298 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:06.399116    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d72jm" podUID="e0651122-b4fd-4a56-8909-53661d776bfc"
Jul 27 17:01:06.399298 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:06.399187    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:06.399298 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:06.399260    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pt78" podUID="0ab2a8bd-0d12-41c0-ae51-b4e3b262a896"
Jul 27 17:01:08.397916 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:08.397881    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:08.398394 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:08.397884    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:08.398394 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:08.398011    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pt78" podUID="0ab2a8bd-0d12-41c0-ae51-b4e3b262a896"
Jul 27 17:01:08.398394 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:08.398022    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:08.398394 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:08.398109    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8jgtw" podUID="16bb5aa2-183e-4742-a424-4ba30c7e3d95"
Jul 27 17:01:08.398394 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:08.398219    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d72jm" podUID="e0651122-b4fd-4a56-8909-53661d776bfc"
Jul 27 17:01:10.397336 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:10.397257    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:10.397776 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:10.397257    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:10.397776 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:10.397389    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pt78" podUID="0ab2a8bd-0d12-41c0-ae51-b4e3b262a896"
Jul 27 17:01:10.397776 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:10.397257    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:10.397776 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:10.397468    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8jgtw" podUID="16bb5aa2-183e-4742-a424-4ba30c7e3d95"
Jul 27 17:01:10.397776 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:10.397558    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d72jm" podUID="e0651122-b4fd-4a56-8909-53661d776bfc"
Jul 27 17:01:12.396925 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:12.396887    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:12.397437 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:12.396887    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:12.397437 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:12.397033    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d72jm" podUID="e0651122-b4fd-4a56-8909-53661d776bfc"
Jul 27 17:01:12.397437 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:12.397058    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:12.397437 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:12.397162    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pt78" podUID="0ab2a8bd-0d12-41c0-ae51-b4e3b262a896"
Jul 27 17:01:12.397437 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:12.397266    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8jgtw" podUID="16bb5aa2-183e-4742-a424-4ba30c7e3d95"
Jul 27 17:01:14.062186 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:14.062100    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtc76\" (UniqueName: \"kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76\") pod \"network-check-target-6pt78\" (UID: \"0ab2a8bd-0d12-41c0-ae51-b4e3b262a896\") " pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:14.062625 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:14.062278    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:01:14.062625 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:14.062297    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:01:14.062625 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:14.062306    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gtc76 for pod openshift-network-diagnostics/network-check-target-6pt78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:14.062625 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:14.062357    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76 podName:0ab2a8bd-0d12-41c0-ae51-b4e3b262a896 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:30.062342396 +0000 UTC m=+34.137606880 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gtc76" (UniqueName: "kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76") pod "network-check-target-6pt78" (UID: "0ab2a8bd-0d12-41c0-ae51-b4e3b262a896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:14.163063 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:14.163032    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret\") pod \"global-pull-secret-syncer-8jgtw\" (UID: \"16bb5aa2-183e-4742-a424-4ba30c7e3d95\") " pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:14.163245 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:14.163100    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs\") pod \"network-metrics-daemon-d72jm\" (UID: \"e0651122-b4fd-4a56-8909-53661d776bfc\") " pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:14.163245 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:14.163209    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:14.163353 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:14.163270    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret podName:16bb5aa2-183e-4742-a424-4ba30c7e3d95 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:30.1632526 +0000 UTC m=+34.238517082 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret") pod "global-pull-secret-syncer-8jgtw" (UID: "16bb5aa2-183e-4742-a424-4ba30c7e3d95") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:14.163353 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:14.163215    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:14.163353 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:14.163354    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs podName:e0651122-b4fd-4a56-8909-53661d776bfc nodeName:}" failed. No retries permitted until 2026-07-27 17:01:30.163337686 +0000 UTC m=+34.238602150 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs") pod "network-metrics-daemon-d72jm" (UID: "e0651122-b4fd-4a56-8909-53661d776bfc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:14.397901 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:14.397815    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:14.397901 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:14.397859    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:14.398122 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:14.397820    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:14.398122 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:14.397950    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8jgtw" podUID="16bb5aa2-183e-4742-a424-4ba30c7e3d95"
Jul 27 17:01:14.398122 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:14.398024    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d72jm" podUID="e0651122-b4fd-4a56-8909-53661d776bfc"
Jul 27 17:01:14.398122 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:14.398095    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pt78" podUID="0ab2a8bd-0d12-41c0-ae51-b4e3b262a896"
Jul 27 17:01:16.397453 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:16.397421    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:16.397918 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:16.397510    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:16.397918 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:16.397526    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pt78" podUID="0ab2a8bd-0d12-41c0-ae51-b4e3b262a896"
Jul 27 17:01:16.397918 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:16.397632    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8jgtw" podUID="16bb5aa2-183e-4742-a424-4ba30c7e3d95"
Jul 27 17:01:16.397918 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:16.397672    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:16.397918 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:16.397763    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d72jm" podUID="e0651122-b4fd-4a56-8909-53661d776bfc"
Jul 27 17:01:18.397194 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.397011    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:18.397678 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.397011    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:18.397678 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:18.397275    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pt78" podUID="0ab2a8bd-0d12-41c0-ae51-b4e3b262a896"
Jul 27 17:01:18.397678 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.397090    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:18.397678 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:18.397440    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d72jm" podUID="e0651122-b4fd-4a56-8909-53661d776bfc"
Jul 27 17:01:18.397678 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:18.397494    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8jgtw" podUID="16bb5aa2-183e-4742-a424-4ba30c7e3d95"
Jul 27 17:01:18.524106 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.524078    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v" event={"ID":"1b606708-d7b9-4eab-8da8-ecfd50f006fb","Type":"ContainerStarted","Data":"6e9deda92368bb9161b58a6ff96e1b7ee3a89a31cb492b7d72532c2e20e43e19"}
Jul 27 17:01:18.525419 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.525401    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vclxn" event={"ID":"2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d","Type":"ContainerStarted","Data":"4e43aa88e8929cbe6c0932533ef5d8ba9f8b823194034032a72494e488f09176"}
Jul 27 17:01:18.526905 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.526884    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8m5b9" event={"ID":"952b4a9c-f2f9-4bf9-a02b-cc426d1995c1","Type":"ContainerStarted","Data":"bfad741aa5a181cee37c7bdabd7b57f03d4aacb9a4541bbc6831a96d38ad475e"}
Jul 27 17:01:18.528336 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.528317    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk" event={"ID":"44c9f115-51d1-4d4a-9a71-0c925017f191","Type":"ContainerStarted","Data":"073616ba5f6f4fa93fc77e32afebc83f55ee01976e92bf676f72144856b84542"}
Jul 27 17:01:18.529734 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.529706    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6kcft" event={"ID":"e02881b5-7256-40d5-9ad4-199b7c349a48","Type":"ContainerStarted","Data":"cdafa6cd83b580fd7b96dc9d32208e4a969c88fc6de082bf2fc694f7e7be8972"}
Jul 27 17:01:18.531188 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.531152    2566 generic.go:356] "Generic (PLEG): container finished" podID="edb57dae-199b-48e9-9a8c-592f3101d307" containerID="fe6373bf538d34eb2c57673638c97ebda4aea61ae679e2fc5862f38048f8fd1d" exitCode=0
Jul 27 17:01:18.531271 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.531195    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mjl8" event={"ID":"edb57dae-199b-48e9-9a8c-592f3101d307","Type":"ContainerDied","Data":"fe6373bf538d34eb2c57673638c97ebda4aea61ae679e2fc5862f38048f8fd1d"}
Jul 27 17:01:18.532771 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.532750    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vh2nd" event={"ID":"10b7f9b7-2929-40bd-8e34-dad3d098187e","Type":"ContainerStarted","Data":"1af3bb4eeafc36defdfd983034985792f9c56b87bad1150be710b163060bb2a4"}
Jul 27 17:01:18.534229 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.534211    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2hn7j" event={"ID":"676386a3-b937-45a3-88e9-54649e0839e0","Type":"ContainerStarted","Data":"e4eaa0c4438705a80b679675e06d394ad718cab2effa63a21a8baa92b48539dd"}
Jul 27 17:01:18.543782 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.543746    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-86.ec2.internal" podStartSLOduration=21.543734004 podStartE2EDuration="21.543734004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:01.507266126 +0000 UTC m=+5.582530613" watchObservedRunningTime="2026-07-27 17:01:18.543734004 +0000 UTC m=+22.618998491"
Jul 27 17:01:18.544111 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.544083    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vclxn" podStartSLOduration=3.884000969 podStartE2EDuration="22.544074138s" totalImagesPullingTime="18.660073169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:18.543512641 +0000 UTC m=+22.618777128" watchObservedRunningTime="2026-07-27 17:01:18.544074138 +0000 UTC m=+22.619338625"
Jul 27 17:01:18.559784 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.559745    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2hn7j" podStartSLOduration=3.907648238 podStartE2EDuration="22.55973127s" totalImagesPullingTime="18.652083032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:18.559453448 +0000 UTC m=+22.634717934" watchObservedRunningTime="2026-07-27 17:01:18.55973127 +0000 UTC m=+22.634995756"
Jul 27 17:01:18.578259 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.578219    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vh2nd" podStartSLOduration=3.927530086 podStartE2EDuration="22.578207568s" totalImagesPullingTime="18.650677482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:18.577665355 +0000 UTC m=+22.652929843" watchObservedRunningTime="2026-07-27 17:01:18.578207568 +0000 UTC m=+22.653472055"
Jul 27 17:01:18.596638 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.596599    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6kcft" podStartSLOduration=3.9407195379999997 podStartE2EDuration="22.596584732s" totalImagesPullingTime="18.655865194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:18.59637925 +0000 UTC m=+22.671643738" watchObservedRunningTime="2026-07-27 17:01:18.596584732 +0000 UTC m=+22.671849219"
Jul 27 17:01:18.621724 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.621687    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8m5b9" podStartSLOduration=3.972295961 podStartE2EDuration="22.621676584s" totalImagesPullingTime="18.649380623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:18.621310432 +0000 UTC m=+22.696574922" watchObservedRunningTime="2026-07-27 17:01:18.621676584 +0000 UTC m=+22.696941072"
Jul 27 17:01:18.690950 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.690828    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vclxn"
Jul 27 17:01:18.691405 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:18.691387    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vclxn"
Jul 27 17:01:19.542702 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:19.542676    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:01:19.543458 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:19.543131    2566 generic.go:356] "Generic (PLEG): container finished" podID="1b606708-d7b9-4eab-8da8-ecfd50f006fb" containerID="fab1235f1008cf2be19bbb05e1a22646c2bf9bed308dd84df7b014d4c2002615" exitCode=1
Jul 27 17:01:19.543458 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:19.543366    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v" event={"ID":"1b606708-d7b9-4eab-8da8-ecfd50f006fb","Type":"ContainerStarted","Data":"5978b156321fc0cbeb156ab671b330e3291da70e463f9404e268fa9f3fe8a61e"}
Jul 27 17:01:19.543583 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:19.543527    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v" event={"ID":"1b606708-d7b9-4eab-8da8-ecfd50f006fb","Type":"ContainerStarted","Data":"d3366004697113df7761b757e293b0df7251f2a7b468ecec665228e128b75d5a"}
Jul 27 17:01:19.543640 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:19.543600    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v" event={"ID":"1b606708-d7b9-4eab-8da8-ecfd50f006fb","Type":"ContainerStarted","Data":"40f917701abaf562c167bd5d688f7f6ca50bb6fb956b2df641c1b607dfdb6498"}
Jul 27 17:01:19.543696 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:19.543681    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v" event={"ID":"1b606708-d7b9-4eab-8da8-ecfd50f006fb","Type":"ContainerStarted","Data":"25ff87b1ea72d71a6b44e1f7e0ebc4511e0ba416c9a27c5b6de4370c99392b12"}
Jul 27 17:01:19.543770 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:19.543748    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v" event={"ID":"1b606708-d7b9-4eab-8da8-ecfd50f006fb","Type":"ContainerDied","Data":"fab1235f1008cf2be19bbb05e1a22646c2bf9bed308dd84df7b014d4c2002615"}
Jul 27 17:01:19.548345 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:19.548324    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 17:01:20.388303 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:20.388198    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T17:01:19.548341229Z","UUID":"6e785e56-2056-445d-9ba1-62fb342b5bb3","Handler":null,"Name":"","Endpoint":""}
Jul 27 17:01:20.389876 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:20.389857    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 17:01:20.390005 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:20.389890    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 17:01:20.396829 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:20.396790    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:20.396937 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:20.396790    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:20.397009 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:20.396916    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pt78" podUID="0ab2a8bd-0d12-41c0-ae51-b4e3b262a896"
Jul 27 17:01:20.397065 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:20.397025    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d72jm" podUID="e0651122-b4fd-4a56-8909-53661d776bfc"
Jul 27 17:01:20.397065 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:20.396790    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:20.397193 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:20.397118    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8jgtw" podUID="16bb5aa2-183e-4742-a424-4ba30c7e3d95"
Jul 27 17:01:20.547843 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:20.547800    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk" event={"ID":"44c9f115-51d1-4d4a-9a71-0c925017f191","Type":"ContainerStarted","Data":"70c2d520718af5d36f07749510afbf25078ac7efa16b5eaf5de9e0c17560336e"}
Jul 27 17:01:20.549575 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:20.549546    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9sh7d" event={"ID":"acad7c3c-4a24-45a5-b9e3-9dbdd6319d02","Type":"ContainerStarted","Data":"883f96bcb82c9994b3e7576761724fec2fae1c44720b0ea08868ec47fff864c7"}
Jul 27 17:01:20.549696 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:20.549582    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 27 17:01:20.567794 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:20.567749    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9sh7d" podStartSLOduration=5.919816831 podStartE2EDuration="24.567729802s" totalImagesPullingTime="18.647912971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:20.56738152 +0000 UTC m=+24.642646007" watchObservedRunningTime="2026-07-27 17:01:20.567729802 +0000 UTC m=+24.642994291"
Jul 27 17:01:21.555802 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:21.555584    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:01:21.556192 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:21.556147    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v" event={"ID":"1b606708-d7b9-4eab-8da8-ecfd50f006fb","Type":"ContainerStarted","Data":"4f813320e045633ef3353a1dee86e12aecec4a95a8a4661e38a9bcd9fb69fae7"}
Jul 27 17:01:21.558124 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:21.558092    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk" event={"ID":"44c9f115-51d1-4d4a-9a71-0c925017f191","Type":"ContainerStarted","Data":"b5b67a53a836434c1a6d4eb63c47e872a0f360d084159291979d82139f57bb1c"}
Jul 27 17:01:21.577614 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:21.577558    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wp9wk" podStartSLOduration=4.053270285 podStartE2EDuration="25.57754326s" totalImagesPullingTime="21.524272975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:00:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:21.577067871 +0000 UTC m=+25.652332349" watchObservedRunningTime="2026-07-27 17:01:21.57754326 +0000 UTC m=+25.652807747"
Jul 27 17:01:22.397886 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:22.397832    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:22.397886 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:22.397863    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:22.398221 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:22.397835    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:22.398221 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:22.397942    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pt78" podUID="0ab2a8bd-0d12-41c0-ae51-b4e3b262a896"
Jul 27 17:01:22.398221 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:22.398039    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8jgtw" podUID="16bb5aa2-183e-4742-a424-4ba30c7e3d95"
Jul 27 17:01:22.398221 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:22.398129    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d72jm" podUID="e0651122-b4fd-4a56-8909-53661d776bfc"
Jul 27 17:01:23.565389 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:23.565256    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:01:23.565950 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:23.565699    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v" event={"ID":"1b606708-d7b9-4eab-8da8-ecfd50f006fb","Type":"ContainerStarted","Data":"60da7debd2220b35e46229a63ca8344c852da4a7a4401f2890b69499eead7cd6"}
Jul 27 17:01:23.565992 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:23.565954    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:01:23.566032 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:23.566002    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:01:23.566064 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:23.566047    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:01:23.566190 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:23.566131    2566 scope.go:117] "RemoveContainer" containerID="fab1235f1008cf2be19bbb05e1a22646c2bf9bed308dd84df7b014d4c2002615"
Jul 27 17:01:23.590406 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:23.590387    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:01:23.590535 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:23.590520    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:01:24.397546 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:24.397507    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:24.397735 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:24.397507    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:24.397735 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:24.397631    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pt78" podUID="0ab2a8bd-0d12-41c0-ae51-b4e3b262a896"
Jul 27 17:01:24.397735 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:24.397521    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:24.397904 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:24.397687    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8jgtw" podUID="16bb5aa2-183e-4742-a424-4ba30c7e3d95"
Jul 27 17:01:24.397904 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:24.397806    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d72jm" podUID="e0651122-b4fd-4a56-8909-53661d776bfc"
Jul 27 17:01:24.570577 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:24.570484    2566 generic.go:356] "Generic (PLEG): container finished" podID="edb57dae-199b-48e9-9a8c-592f3101d307" containerID="7a871c844c7c91608d821a0f191475c297de664f80bcb6ea1be21b5b8f676b9f" exitCode=0
Jul 27 17:01:24.571107 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:24.570565    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mjl8" event={"ID":"edb57dae-199b-48e9-9a8c-592f3101d307","Type":"ContainerDied","Data":"7a871c844c7c91608d821a0f191475c297de664f80bcb6ea1be21b5b8f676b9f"}
Jul 27 17:01:24.575254 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:24.575233    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:01:24.575789 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:24.575752    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v" event={"ID":"1b606708-d7b9-4eab-8da8-ecfd50f006fb","Type":"ContainerStarted","Data":"92312d9b3c713e3a3e98d8ec27ae8be4c2a78aeac8a255c42eed8156a1c202c9"}
Jul 27 17:01:24.622917 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:24.622852    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v" podStartSLOduration=9.670570119 podStartE2EDuration="28.622834345s" totalImagesPullingTime="18.952264226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:24.622004946 +0000 UTC m=+28.697269434" watchObservedRunningTime="2026-07-27 17:01:24.622834345 +0000 UTC m=+28.698098836"
Jul 27 17:01:24.815843 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:24.815813    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vclxn"
Jul 27 17:01:24.816076 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:24.815953    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 27 17:01:24.816345 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:24.816326    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vclxn"
Jul 27 17:01:25.591554 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:25.591500    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8jgtw"]
Jul 27 17:01:25.592009 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:25.591647    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:25.592009 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:25.591757    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8jgtw" podUID="16bb5aa2-183e-4742-a424-4ba30c7e3d95"
Jul 27 17:01:25.594717 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:25.594691    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6pt78"]
Jul 27 17:01:25.594841 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:25.594776    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:25.594885 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:25.594838    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pt78" podUID="0ab2a8bd-0d12-41c0-ae51-b4e3b262a896"
Jul 27 17:01:25.597528 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:25.597507    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d72jm"]
Jul 27 17:01:25.597626 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:25.597578    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:25.597660 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:25.597644    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d72jm" podUID="e0651122-b4fd-4a56-8909-53661d776bfc"
Jul 27 17:01:26.582090 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:26.582060    2566 generic.go:356] "Generic (PLEG): container finished" podID="edb57dae-199b-48e9-9a8c-592f3101d307" containerID="4f3ad34347c3816caa28b73350b337f589eba72c1ec7d032a1543948f09c657c" exitCode=0
Jul 27 17:01:26.582261 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:26.582115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mjl8" event={"ID":"edb57dae-199b-48e9-9a8c-592f3101d307","Type":"ContainerDied","Data":"4f3ad34347c3816caa28b73350b337f589eba72c1ec7d032a1543948f09c657c"}
Jul 27 17:01:27.397658 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:27.397487    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:27.398121 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:27.397554    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:27.398121 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:27.397748    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8jgtw" podUID="16bb5aa2-183e-4742-a424-4ba30c7e3d95"
Jul 27 17:01:27.398121 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:27.397565    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:27.398121 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:27.397810    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d72jm" podUID="e0651122-b4fd-4a56-8909-53661d776bfc"
Jul 27 17:01:27.398121 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:27.397890    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pt78" podUID="0ab2a8bd-0d12-41c0-ae51-b4e3b262a896"
Jul 27 17:01:28.588185 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:28.588125    2566 generic.go:356] "Generic (PLEG): container finished" podID="edb57dae-199b-48e9-9a8c-592f3101d307" containerID="ccae8113a768ce9262ac3cd6337ade7a5613c887974b8e8c662ae28bd009e52b" exitCode=0
Jul 27 17:01:28.588841 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:28.588207    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mjl8" event={"ID":"edb57dae-199b-48e9-9a8c-592f3101d307","Type":"ContainerDied","Data":"ccae8113a768ce9262ac3cd6337ade7a5613c887974b8e8c662ae28bd009e52b"}
Jul 27 17:01:29.397204 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:29.397095    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:29.397204 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:29.397108    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:29.397408 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:29.397238    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d72jm" podUID="e0651122-b4fd-4a56-8909-53661d776bfc"
Jul 27 17:01:29.397408 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:29.397278    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:29.397408 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:29.397356    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8jgtw" podUID="16bb5aa2-183e-4742-a424-4ba30c7e3d95"
Jul 27 17:01:29.397496 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:29.397426    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6pt78" podUID="0ab2a8bd-0d12-41c0-ae51-b4e3b262a896"
Jul 27 17:01:30.081472 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.081437    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtc76\" (UniqueName: \"kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76\") pod \"network-check-target-6pt78\" (UID: \"0ab2a8bd-0d12-41c0-ae51-b4e3b262a896\") " pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:30.082068 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:30.081573    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:01:30.082068 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:30.081588    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:01:30.082068 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:30.081598    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gtc76 for pod openshift-network-diagnostics/network-check-target-6pt78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:30.082068 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:30.081660    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76 podName:0ab2a8bd-0d12-41c0-ae51-b4e3b262a896 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:02.081644199 +0000 UTC m=+66.156908679 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gtc76" (UniqueName: "kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76") pod "network-check-target-6pt78" (UID: "0ab2a8bd-0d12-41c0-ae51-b4e3b262a896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:30.182225 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.182189    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret\") pod \"global-pull-secret-syncer-8jgtw\" (UID: \"16bb5aa2-183e-4742-a424-4ba30c7e3d95\") " pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:30.182408 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.182268    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs\") pod \"network-metrics-daemon-d72jm\" (UID: \"e0651122-b4fd-4a56-8909-53661d776bfc\") " pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:30.182408 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:30.182362    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:30.182518 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:30.182419    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:30.182518 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:30.182439    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret podName:16bb5aa2-183e-4742-a424-4ba30c7e3d95 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:02.18241742 +0000 UTC m=+66.257681885 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret") pod "global-pull-secret-syncer-8jgtw" (UID: "16bb5aa2-183e-4742-a424-4ba30c7e3d95") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:30.182518 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:30.182490    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs podName:e0651122-b4fd-4a56-8909-53661d776bfc nodeName:}" failed. No retries permitted until 2026-07-27 17:02:02.182469862 +0000 UTC m=+66.257734333 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs") pod "network-metrics-daemon-d72jm" (UID: "e0651122-b4fd-4a56-8909-53661d776bfc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:30.702982 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.702955    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-86.ec2.internal" event="NodeReady"
Jul 27 17:01:30.703125 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.703073    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 17:01:30.748065 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.748021    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6cf44c948f-tc7r6"]
Jul 27 17:01:30.751836 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.751815    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.754304 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.754276    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 17:01:30.754424 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.754287    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-64scz\""
Jul 27 17:01:30.754628 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.754610    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 17:01:30.754720 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.754658    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 17:01:30.761118 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.761076    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-prbcj"]
Jul 27 17:01:30.764269 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.764236    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 17:01:30.765130 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.765112    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-prbcj"
Jul 27 17:01:30.766586 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.766560    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cf44c948f-tc7r6"]
Jul 27 17:01:30.767571 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.767514    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5sbpw\""
Jul 27 17:01:30.767680 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.767571    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 17:01:30.768515 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.768497    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 17:01:30.769854 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.769646    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6tmv2"]
Jul 27 17:01:30.775509 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.775488    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6tmv2"
Jul 27 17:01:30.776920 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.776856    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-prbcj"]
Jul 27 17:01:30.778148 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.778105    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dmcwl\""
Jul 27 17:01:30.778492 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.778468    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 17:01:30.778492 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.778482    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 17:01:30.778642 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.778571    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 17:01:30.788184 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.788142    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6tmv2"]
Jul 27 17:01:30.888469 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.888390    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npm6d\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-kube-api-access-npm6d\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.888469 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.888422    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9zr5\" (UniqueName: \"kubernetes.io/projected/acde9127-43b5-46ed-9a95-515b25b0faf7-kube-api-access-x9zr5\") pod \"dns-default-prbcj\" (UID: \"acde9127-43b5-46ed-9a95-515b25b0faf7\") " pod="openshift-dns/dns-default-prbcj"
Jul 27 17:01:30.888469 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.888447    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79213db4-70fd-4b19-aaf6-d052590ae819-installation-pull-secrets\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.888469 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.888468    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-bound-sa-token\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.888889 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.888579    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msmwp\" (UniqueName: \"kubernetes.io/projected/9d9ff7dd-7fa8-4a73-9b78-76fdd0531ef7-kube-api-access-msmwp\") pod \"ingress-canary-6tmv2\" (UID: \"9d9ff7dd-7fa8-4a73-9b78-76fdd0531ef7\") " pod="openshift-ingress-canary/ingress-canary-6tmv2"
Jul 27 17:01:30.888889 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.888615    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.888889 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.888641    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79213db4-70fd-4b19-aaf6-d052590ae819-trusted-ca\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.888889 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.888682    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls\") pod \"dns-default-prbcj\" (UID: \"acde9127-43b5-46ed-9a95-515b25b0faf7\") " pod="openshift-dns/dns-default-prbcj"
Jul 27 17:01:30.888889 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.888717    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79213db4-70fd-4b19-aaf6-d052590ae819-image-registry-private-configuration\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.888889 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.888740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79213db4-70fd-4b19-aaf6-d052590ae819-registry-certificates\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.888889 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.888762    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acde9127-43b5-46ed-9a95-515b25b0faf7-tmp-dir\") pod \"dns-default-prbcj\" (UID: \"acde9127-43b5-46ed-9a95-515b25b0faf7\") " pod="openshift-dns/dns-default-prbcj"
Jul 27 17:01:30.888889 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.888786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acde9127-43b5-46ed-9a95-515b25b0faf7-config-volume\") pod \"dns-default-prbcj\" (UID: \"acde9127-43b5-46ed-9a95-515b25b0faf7\") " pod="openshift-dns/dns-default-prbcj"
Jul 27 17:01:30.888889 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.888859    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79213db4-70fd-4b19-aaf6-d052590ae819-ca-trust-extracted\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.888889 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.888889    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d9ff7dd-7fa8-4a73-9b78-76fdd0531ef7-cert\") pod \"ingress-canary-6tmv2\" (UID: \"9d9ff7dd-7fa8-4a73-9b78-76fdd0531ef7\") " pod="openshift-ingress-canary/ingress-canary-6tmv2"
Jul 27 17:01:30.989838 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.989785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msmwp\" (UniqueName: \"kubernetes.io/projected/9d9ff7dd-7fa8-4a73-9b78-76fdd0531ef7-kube-api-access-msmwp\") pod \"ingress-canary-6tmv2\" (UID: \"9d9ff7dd-7fa8-4a73-9b78-76fdd0531ef7\") " pod="openshift-ingress-canary/ingress-canary-6tmv2"
Jul 27 17:01:30.989838 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.989841    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.990144 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.989864    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79213db4-70fd-4b19-aaf6-d052590ae819-trusted-ca\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.990144 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.989901    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls\") pod \"dns-default-prbcj\" (UID: \"acde9127-43b5-46ed-9a95-515b25b0faf7\") " pod="openshift-dns/dns-default-prbcj"
Jul 27 17:01:30.990144 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.989932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79213db4-70fd-4b19-aaf6-d052590ae819-image-registry-private-configuration\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.990144 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.989951    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79213db4-70fd-4b19-aaf6-d052590ae819-registry-certificates\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.990144 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.989970    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acde9127-43b5-46ed-9a95-515b25b0faf7-tmp-dir\") pod \"dns-default-prbcj\" (UID: \"acde9127-43b5-46ed-9a95-515b25b0faf7\") " pod="openshift-dns/dns-default-prbcj"
Jul 27 17:01:30.990144 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.989992    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acde9127-43b5-46ed-9a95-515b25b0faf7-config-volume\") pod \"dns-default-prbcj\" (UID: \"acde9127-43b5-46ed-9a95-515b25b0faf7\") " pod="openshift-dns/dns-default-prbcj"
Jul 27 17:01:30.990144 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:30.990021    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:01:30.990144 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.990036    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79213db4-70fd-4b19-aaf6-d052590ae819-ca-trust-extracted\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.990144 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:30.990042    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cf44c948f-tc7r6: secret "image-registry-tls" not found
Jul 27 17:01:30.990144 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.990051    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d9ff7dd-7fa8-4a73-9b78-76fdd0531ef7-cert\") pod \"ingress-canary-6tmv2\" (UID: \"9d9ff7dd-7fa8-4a73-9b78-76fdd0531ef7\") " pod="openshift-ingress-canary/ingress-canary-6tmv2"
Jul 27 17:01:30.990144 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.990077    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npm6d\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-kube-api-access-npm6d\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.990144 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.990099    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9zr5\" (UniqueName: \"kubernetes.io/projected/acde9127-43b5-46ed-9a95-515b25b0faf7-kube-api-access-x9zr5\") pod \"dns-default-prbcj\" (UID: \"acde9127-43b5-46ed-9a95-515b25b0faf7\") " pod="openshift-dns/dns-default-prbcj"
Jul 27 17:01:30.990144 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:30.990118    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls podName:79213db4-70fd-4b19-aaf6-d052590ae819 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:31.490096321 +0000 UTC m=+35.565360799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls") pod "image-registry-6cf44c948f-tc7r6" (UID: "79213db4-70fd-4b19-aaf6-d052590ae819") : secret "image-registry-tls" not found
Jul 27 17:01:30.990971 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.990186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79213db4-70fd-4b19-aaf6-d052590ae819-installation-pull-secrets\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.990971 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.990217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-bound-sa-token\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.990971 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:30.990356    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:01:30.990971 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:30.990469    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls podName:acde9127-43b5-46ed-9a95-515b25b0faf7 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:31.490451721 +0000 UTC m=+35.565716190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls") pod "dns-default-prbcj" (UID: "acde9127-43b5-46ed-9a95-515b25b0faf7") : secret "dns-default-metrics-tls" not found
Jul 27 17:01:30.990971 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.990594    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acde9127-43b5-46ed-9a95-515b25b0faf7-tmp-dir\") pod \"dns-default-prbcj\" (UID: \"acde9127-43b5-46ed-9a95-515b25b0faf7\") " pod="openshift-dns/dns-default-prbcj"
Jul 27 17:01:30.990971 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.990642    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79213db4-70fd-4b19-aaf6-d052590ae819-ca-trust-extracted\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.991373 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.991182    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79213db4-70fd-4b19-aaf6-d052590ae819-registry-certificates\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.991373 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.991229    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acde9127-43b5-46ed-9a95-515b25b0faf7-config-volume\") pod \"dns-default-prbcj\" (UID: \"acde9127-43b5-46ed-9a95-515b25b0faf7\") " pod="openshift-dns/dns-default-prbcj"
Jul 27 17:01:30.991483 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.991379    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79213db4-70fd-4b19-aaf6-d052590ae819-trusted-ca\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.995249 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.995201    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79213db4-70fd-4b19-aaf6-d052590ae819-image-registry-private-configuration\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.995249 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.995226    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79213db4-70fd-4b19-aaf6-d052590ae819-installation-pull-secrets\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.995424 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.995260    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d9ff7dd-7fa8-4a73-9b78-76fdd0531ef7-cert\") pod \"ingress-canary-6tmv2\" (UID: \"9d9ff7dd-7fa8-4a73-9b78-76fdd0531ef7\") " pod="openshift-ingress-canary/ingress-canary-6tmv2"
Jul 27 17:01:30.999214 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.999191    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msmwp\" (UniqueName: \"kubernetes.io/projected/9d9ff7dd-7fa8-4a73-9b78-76fdd0531ef7-kube-api-access-msmwp\") pod \"ingress-canary-6tmv2\" (UID: \"9d9ff7dd-7fa8-4a73-9b78-76fdd0531ef7\") " pod="openshift-ingress-canary/ingress-canary-6tmv2"
Jul 27 17:01:30.999397 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.999377    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9zr5\" (UniqueName: \"kubernetes.io/projected/acde9127-43b5-46ed-9a95-515b25b0faf7-kube-api-access-x9zr5\") pod \"dns-default-prbcj\" (UID: \"acde9127-43b5-46ed-9a95-515b25b0faf7\") " pod="openshift-dns/dns-default-prbcj"
Jul 27 17:01:30.999646 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.999626    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-bound-sa-token\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:30.999707 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:30.999665    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npm6d\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-kube-api-access-npm6d\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:31.088559 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:31.088526    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6tmv2"
Jul 27 17:01:31.243676 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:31.243439    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6tmv2"]
Jul 27 17:01:31.247514 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:01:31.247473    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9ff7dd_7fa8_4a73_9b78_76fdd0531ef7.slice/crio-f3f1991bcc2e15442b28d7bdc07567d24c6d0109a521f4db224735384fc2cfb3 WatchSource:0}: Error finding container f3f1991bcc2e15442b28d7bdc07567d24c6d0109a521f4db224735384fc2cfb3: Status 404 returned error can't find the container with id f3f1991bcc2e15442b28d7bdc07567d24c6d0109a521f4db224735384fc2cfb3
Jul 27 17:01:31.397156 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:31.397080    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:01:31.397156 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:31.397109    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:01:31.397156 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:31.397109    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:01:31.399943 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:31.399921    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 17:01:31.400051 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:31.399950    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 17:01:31.400051 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:31.400035    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 17:01:31.400148 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:31.399925    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 17:01:31.400217 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:31.400195    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c7drs\""
Jul 27 17:01:31.400324 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:31.400305    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mxtqx\""
Jul 27 17:01:31.494100 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:31.494063    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:31.494303 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:31.494123    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls\") pod \"dns-default-prbcj\" (UID: \"acde9127-43b5-46ed-9a95-515b25b0faf7\") " pod="openshift-dns/dns-default-prbcj"
Jul 27 17:01:31.494303 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:31.494239    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:01:31.494303 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:31.494260    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cf44c948f-tc7r6: secret "image-registry-tls" not found
Jul 27 17:01:31.494461 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:31.494341    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls podName:79213db4-70fd-4b19-aaf6-d052590ae819 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:32.494320337 +0000 UTC m=+36.569584821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls") pod "image-registry-6cf44c948f-tc7r6" (UID: "79213db4-70fd-4b19-aaf6-d052590ae819") : secret "image-registry-tls" not found
Jul 27 17:01:31.494461 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:31.494426    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:01:31.494556 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:31.494486    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls podName:acde9127-43b5-46ed-9a95-515b25b0faf7 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:32.49447381 +0000 UTC m=+36.569738275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls") pod "dns-default-prbcj" (UID: "acde9127-43b5-46ed-9a95-515b25b0faf7") : secret "dns-default-metrics-tls" not found
Jul 27 17:01:31.595893 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:31.595853    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6tmv2" event={"ID":"9d9ff7dd-7fa8-4a73-9b78-76fdd0531ef7","Type":"ContainerStarted","Data":"f3f1991bcc2e15442b28d7bdc07567d24c6d0109a521f4db224735384fc2cfb3"}
Jul 27 17:01:32.503816 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:32.503775    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:32.504252 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:32.503831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls\") pod \"dns-default-prbcj\" (UID: \"acde9127-43b5-46ed-9a95-515b25b0faf7\") " pod="openshift-dns/dns-default-prbcj"
Jul 27 17:01:32.504252 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:32.503930    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:01:32.504252 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:32.503950    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cf44c948f-tc7r6: secret "image-registry-tls" not found
Jul 27 17:01:32.504252 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:32.503969    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:01:32.504252 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:32.504012    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls podName:79213db4-70fd-4b19-aaf6-d052590ae819 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:34.503991957 +0000 UTC m=+38.579256448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls") pod "image-registry-6cf44c948f-tc7r6" (UID: "79213db4-70fd-4b19-aaf6-d052590ae819") : secret "image-registry-tls" not found
Jul 27 17:01:32.504252 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:32.504031    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls podName:acde9127-43b5-46ed-9a95-515b25b0faf7 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:34.504024978 +0000 UTC m=+38.579289443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls") pod "dns-default-prbcj" (UID: "acde9127-43b5-46ed-9a95-515b25b0faf7") : secret "dns-default-metrics-tls" not found
Jul 27 17:01:32.648132 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:32.648097    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vclxn_2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d/konnectivity-agent/0.log" limit=1024
Jul 27 17:01:32.856017 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:32.855916    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vclxn_2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d/konnectivity-agent/0.log" limit=1024
Jul 27 17:01:32.967162 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:32.967121    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vclxn_2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d/konnectivity-agent/0.log" limit=1024
Jul 27 17:01:33.066728 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:33.066693    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vclxn_2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d/konnectivity-agent/0.log" limit=1024
Jul 27 17:01:33.602829 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:33.602750    2566 generic.go:356] "Generic (PLEG): container finished" podID="edb57dae-199b-48e9-9a8c-592f3101d307" containerID="504d8ad794623d32d8e29a653e02b39bf00d63db85ba65ba5fd59c411c6bcf58" exitCode=0
Jul 27 17:01:33.602829 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:33.602804    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mjl8" event={"ID":"edb57dae-199b-48e9-9a8c-592f3101d307","Type":"ContainerDied","Data":"504d8ad794623d32d8e29a653e02b39bf00d63db85ba65ba5fd59c411c6bcf58"}
Jul 27 17:01:34.520736 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:34.520695    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:34.520904 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:34.520763    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls\") pod \"dns-default-prbcj\" (UID: \"acde9127-43b5-46ed-9a95-515b25b0faf7\") " pod="openshift-dns/dns-default-prbcj"
Jul 27 17:01:34.520904 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:34.520843    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:01:34.520904 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:34.520860    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cf44c948f-tc7r6: secret "image-registry-tls" not found
Jul 27 17:01:34.521060 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:34.520913    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls podName:79213db4-70fd-4b19-aaf6-d052590ae819 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:38.520899134 +0000 UTC m=+42.596163599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls") pod "image-registry-6cf44c948f-tc7r6" (UID: "79213db4-70fd-4b19-aaf6-d052590ae819") : secret "image-registry-tls" not found
Jul 27 17:01:34.521060 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:34.520922    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:01:34.521060 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:34.520987    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls podName:acde9127-43b5-46ed-9a95-515b25b0faf7 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:38.520969884 +0000 UTC m=+42.596234365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls") pod "dns-default-prbcj" (UID: "acde9127-43b5-46ed-9a95-515b25b0faf7") : secret "dns-default-metrics-tls" not found
Jul 27 17:01:34.609620 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:34.609569    2566 generic.go:356] "Generic (PLEG): container finished" podID="edb57dae-199b-48e9-9a8c-592f3101d307" containerID="c7c9dcf315924714eff8917d1337a82d55e7d42522b660e172b4116fc5873c28" exitCode=0
Jul 27 17:01:34.610060 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:34.609659    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mjl8" event={"ID":"edb57dae-199b-48e9-9a8c-592f3101d307","Type":"ContainerDied","Data":"c7c9dcf315924714eff8917d1337a82d55e7d42522b660e172b4116fc5873c28"}
Jul 27 17:01:35.614111 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:35.613861    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6tmv2" event={"ID":"9d9ff7dd-7fa8-4a73-9b78-76fdd0531ef7","Type":"ContainerStarted","Data":"39d444e01eebc1d54e0343ad918621189f59270542c6c1819e81fb6f0b308d18"}
Jul 27 17:01:35.617443 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:35.617416    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mjl8" event={"ID":"edb57dae-199b-48e9-9a8c-592f3101d307","Type":"ContainerStarted","Data":"77491159dbf54cd2fefe747cc2b2134e1f7c940637fa8f08d69542e1c6db0e71"}
Jul 27 17:01:35.630744 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:35.630693    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6tmv2" podStartSLOduration=1.472806263 podStartE2EDuration="5.630677457s" totalImagesPullingTime="4.157871194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:35.630086393 +0000 UTC m=+39.705350881" watchObservedRunningTime="2026-07-27 17:01:35.630677457 +0000 UTC m=+39.705941943"
Jul 27 17:01:35.653839 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:35.653756    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2mjl8" podStartSLOduration=12.954403683 podStartE2EDuration="39.653738618s" totalImagesPullingTime="26.699334935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:00:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:35.652732549 +0000 UTC m=+39.727997037" watchObservedRunningTime="2026-07-27 17:01:35.653738618 +0000 UTC m=+39.729003107"
Jul 27 17:01:36.122740 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:36.122700    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vclxn_2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d/konnectivity-agent/0.log" limit=1024
Jul 27 17:01:38.550810 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:38.550771    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:38.551307 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:38.550822    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls\") pod \"dns-default-prbcj\" (UID: \"acde9127-43b5-46ed-9a95-515b25b0faf7\") " pod="openshift-dns/dns-default-prbcj"
Jul 27 17:01:38.551307 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:38.550920    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:01:38.551307 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:38.550926    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:01:38.551307 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:38.550942    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cf44c948f-tc7r6: secret "image-registry-tls" not found
Jul 27 17:01:38.551307 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:38.550968    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls podName:acde9127-43b5-46ed-9a95-515b25b0faf7 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:46.550956255 +0000 UTC m=+50.626220720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls") pod "dns-default-prbcj" (UID: "acde9127-43b5-46ed-9a95-515b25b0faf7") : secret "dns-default-metrics-tls" not found
Jul 27 17:01:38.551307 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:38.550986    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls podName:79213db4-70fd-4b19-aaf6-d052590ae819 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:46.550974403 +0000 UTC m=+50.626238884 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls") pod "image-registry-6cf44c948f-tc7r6" (UID: "79213db4-70fd-4b19-aaf6-d052590ae819") : secret "image-registry-tls" not found
Jul 27 17:01:39.331318 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:39.331285    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vclxn_2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d/konnectivity-agent/0.log" limit=1024
Jul 27 17:01:40.157738 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:40.157698    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vclxn_2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d/konnectivity-agent/0.log" limit=1024
Jul 27 17:01:40.215670 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:40.215633    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vclxn_2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d/konnectivity-agent/0.log" limit=1024
Jul 27 17:01:45.071460 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:45.071414    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vclxn_2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d/konnectivity-agent/0.log" limit=1024
Jul 27 17:01:45.171393 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:45.171351    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vclxn_2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d/konnectivity-agent/0.log" limit=1024
Jul 27 17:01:46.613103 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:46.613065    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:01:46.613103 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:46.613109    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls\") pod \"dns-default-prbcj\" (UID: \"acde9127-43b5-46ed-9a95-515b25b0faf7\") " pod="openshift-dns/dns-default-prbcj"
Jul 27 17:01:46.613581 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:46.613215    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:01:46.613581 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:46.613237    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:01:46.613581 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:46.613259    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cf44c948f-tc7r6: secret "image-registry-tls" not found
Jul 27 17:01:46.613581 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:46.613268    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls podName:acde9127-43b5-46ed-9a95-515b25b0faf7 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:02.61325386 +0000 UTC m=+66.688518324 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls") pod "dns-default-prbcj" (UID: "acde9127-43b5-46ed-9a95-515b25b0faf7") : secret "dns-default-metrics-tls" not found
Jul 27 17:01:46.613581 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:01:46.613327    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls podName:79213db4-70fd-4b19-aaf6-d052590ae819 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:02.613308152 +0000 UTC m=+66.688572641 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls") pod "image-registry-6cf44c948f-tc7r6" (UID: "79213db4-70fd-4b19-aaf6-d052590ae819") : secret "image-registry-tls" not found
Jul 27 17:01:55.587863 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:01:55.587832    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m9k9v"
Jul 27 17:02:02.116991 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:02.116936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtc76\" (UniqueName: \"kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76\") pod \"network-check-target-6pt78\" (UID: \"0ab2a8bd-0d12-41c0-ae51-b4e3b262a896\") " pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:02:02.119884 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:02.119862    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 17:02:02.129731 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:02.129706    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 17:02:02.142015 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:02.141984    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtc76\" (UniqueName: \"kubernetes.io/projected/0ab2a8bd-0d12-41c0-ae51-b4e3b262a896-kube-api-access-gtc76\") pod \"network-check-target-6pt78\" (UID: \"0ab2a8bd-0d12-41c0-ae51-b4e3b262a896\") " pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:02:02.218015 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:02.217967    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs\") pod \"network-metrics-daemon-d72jm\" (UID: \"e0651122-b4fd-4a56-8909-53661d776bfc\") " pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:02:02.218015 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:02.218034    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret\") pod \"global-pull-secret-syncer-8jgtw\" (UID: \"16bb5aa2-183e-4742-a424-4ba30c7e3d95\") " pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:02:02.220571 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:02.220553    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 17:02:02.220751 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:02.220734    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 17:02:02.228728 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:02.228709    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 17:02:02.228786 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:02.228775    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs podName:e0651122-b4fd-4a56-8909-53661d776bfc nodeName:}" failed. No retries permitted until 2026-07-27 17:03:06.22875919 +0000 UTC m=+130.304023656 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs") pod "network-metrics-daemon-d72jm" (UID: "e0651122-b4fd-4a56-8909-53661d776bfc") : secret "metrics-daemon-secret" not found
Jul 27 17:02:02.230844 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:02.230813    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/16bb5aa2-183e-4742-a424-4ba30c7e3d95-original-pull-secret\") pod \"global-pull-secret-syncer-8jgtw\" (UID: \"16bb5aa2-183e-4742-a424-4ba30c7e3d95\") " pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:02:02.321852 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:02.321824    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c7drs\""
Jul 27 17:02:02.323991 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:02.323976    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8jgtw"
Jul 27 17:02:02.329809 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:02.329781    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:02:02.452075 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:02.452044    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8jgtw"]
Jul 27 17:02:02.455595 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:02:02.455566    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16bb5aa2_183e_4742_a424_4ba30c7e3d95.slice/crio-41ca02f603996e6119218aae8df0f3f636f2d3d2b14e85099c4fca946a6ea1e0 WatchSource:0}: Error finding container 41ca02f603996e6119218aae8df0f3f636f2d3d2b14e85099c4fca946a6ea1e0: Status 404 returned error can't find the container with id 41ca02f603996e6119218aae8df0f3f636f2d3d2b14e85099c4fca946a6ea1e0
Jul 27 17:02:02.467891 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:02.467867    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6pt78"]
Jul 27 17:02:02.472685 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:02:02.472662    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab2a8bd_0d12_41c0_ae51_b4e3b262a896.slice/crio-8edeb6717e88a3f0ae293b89241fa98c44a328cbff8e11bcde108395d52a7dfc WatchSource:0}: Error finding container 8edeb6717e88a3f0ae293b89241fa98c44a328cbff8e11bcde108395d52a7dfc: Status 404 returned error can't find the container with id 8edeb6717e88a3f0ae293b89241fa98c44a328cbff8e11bcde108395d52a7dfc
Jul 27 17:02:02.620344 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:02.620310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:02:02.620344 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:02.620353    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls\") pod \"dns-default-prbcj\" (UID: \"acde9127-43b5-46ed-9a95-515b25b0faf7\") " pod="openshift-dns/dns-default-prbcj"
Jul 27 17:02:02.620579 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:02.620465    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:02:02.620579 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:02.620491    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cf44c948f-tc7r6: secret "image-registry-tls" not found
Jul 27 17:02:02.620579 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:02.620541    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls podName:79213db4-70fd-4b19-aaf6-d052590ae819 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:34.620526516 +0000 UTC m=+98.695790986 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls") pod "image-registry-6cf44c948f-tc7r6" (UID: "79213db4-70fd-4b19-aaf6-d052590ae819") : secret "image-registry-tls" not found
Jul 27 17:02:02.620579 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:02.620474    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:02:02.620708 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:02.620596    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls podName:acde9127-43b5-46ed-9a95-515b25b0faf7 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:34.620584861 +0000 UTC m=+98.695849326 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls") pod "dns-default-prbcj" (UID: "acde9127-43b5-46ed-9a95-515b25b0faf7") : secret "dns-default-metrics-tls" not found
Jul 27 17:02:02.680201 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:02.680102    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8jgtw" event={"ID":"16bb5aa2-183e-4742-a424-4ba30c7e3d95","Type":"ContainerStarted","Data":"41ca02f603996e6119218aae8df0f3f636f2d3d2b14e85099c4fca946a6ea1e0"}
Jul 27 17:02:02.681211 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:02.681189    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6pt78" event={"ID":"0ab2a8bd-0d12-41c0-ae51-b4e3b262a896","Type":"ContainerStarted","Data":"8edeb6717e88a3f0ae293b89241fa98c44a328cbff8e11bcde108395d52a7dfc"}
Jul 27 17:02:07.697539 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:07.697492    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8jgtw" event={"ID":"16bb5aa2-183e-4742-a424-4ba30c7e3d95","Type":"ContainerStarted","Data":"8be182160029f8ca3d5c64184095d545e63e808502b6cfd43e5ed5624c65998c"}
Jul 27 17:02:07.698959 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:07.698935    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6pt78" event={"ID":"0ab2a8bd-0d12-41c0-ae51-b4e3b262a896","Type":"ContainerStarted","Data":"f7714e8d25aafe4e5c9830dc7897f1271aa7050d7111978ce7452a9d49c938b7"}
Jul 27 17:02:07.699095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:07.699083    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:02:07.714391 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:07.714343    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8jgtw" podStartSLOduration=66.69832157 podStartE2EDuration="1m11.714329613s" totalImagesPullingTime="5.016008043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:07.713798121 +0000 UTC m=+71.789062609" watchObservedRunningTime="2026-07-27 17:02:07.714329613 +0000 UTC m=+71.789594100"
Jul 27 17:02:07.732025 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:07.731951    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6pt78" podStartSLOduration=66.736800989 podStartE2EDuration="1m11.731937594s" totalImagesPullingTime="4.995136605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:07.731838294 +0000 UTC m=+71.807102783" watchObservedRunningTime="2026-07-27 17:02:07.731937594 +0000 UTC m=+71.807202080"
Jul 27 17:02:18.043484 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.043448    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"]
Jul 27 17:02:18.046162 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.046142    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-2vf8b"]
Jul 27 17:02:18.046289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.046265    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"
Jul 27 17:02:18.048800 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.048778    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.050055 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.050036    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 17:02:18.050226 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.050209    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 27 17:02:18.050277 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.050235    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 27 17:02:18.050277 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.050268    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 17:02:18.050350 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.050282    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vww87\""
Jul 27 17:02:18.051090 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.051075    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 27 17:02:18.051180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.051097    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 27 17:02:18.051245 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.051192    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 17:02:18.051245 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.051235    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 17:02:18.051346 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.051265    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rz6kc\""
Jul 27 17:02:18.057444 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.057319    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"]
Jul 27 17:02:18.058108 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.058091    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 27 17:02:18.058416 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.058398    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-2vf8b"]
Jul 27 17:02:18.132967 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.132930    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ttxd\" (UniqueName: \"kubernetes.io/projected/50804eac-c593-411c-9aaa-f4a3742cb1fe-kube-api-access-7ttxd\") pod \"cluster-monitoring-operator-548b489cb8-dzpt4\" (UID: \"50804eac-c593-411c-9aaa-f4a3742cb1fe\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"
Jul 27 17:02:18.133161 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.132974    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dzpt4\" (UID: \"50804eac-c593-411c-9aaa-f4a3742cb1fe\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"
Jul 27 17:02:18.133161 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.133008    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6p2r\" (UniqueName: \"kubernetes.io/projected/9a26877e-9895-4df2-b890-bf52e79a3239-kube-api-access-h6p2r\") pod \"insights-operator-5885fdb759-2vf8b\" (UID: \"9a26877e-9895-4df2-b890-bf52e79a3239\") " pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.133161 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.133070    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a26877e-9895-4df2-b890-bf52e79a3239-tmp\") pod \"insights-operator-5885fdb759-2vf8b\" (UID: \"9a26877e-9895-4df2-b890-bf52e79a3239\") " pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.133161 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.133094    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9a26877e-9895-4df2-b890-bf52e79a3239-snapshots\") pod \"insights-operator-5885fdb759-2vf8b\" (UID: \"9a26877e-9895-4df2-b890-bf52e79a3239\") " pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.133161 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.133128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a26877e-9895-4df2-b890-bf52e79a3239-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-2vf8b\" (UID: \"9a26877e-9895-4df2-b890-bf52e79a3239\") " pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.133161 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.133149    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a26877e-9895-4df2-b890-bf52e79a3239-service-ca-bundle\") pod \"insights-operator-5885fdb759-2vf8b\" (UID: \"9a26877e-9895-4df2-b890-bf52e79a3239\") " pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.133438 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.133265    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a26877e-9895-4df2-b890-bf52e79a3239-serving-cert\") pod \"insights-operator-5885fdb759-2vf8b\" (UID: \"9a26877e-9895-4df2-b890-bf52e79a3239\") " pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.133438 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.133320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/50804eac-c593-411c-9aaa-f4a3742cb1fe-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-dzpt4\" (UID: \"50804eac-c593-411c-9aaa-f4a3742cb1fe\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"
Jul 27 17:02:18.144396 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.144371    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g"]
Jul 27 17:02:18.147380 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.147364    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g"
Jul 27 17:02:18.149641 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.149621    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zrqln\""
Jul 27 17:02:18.150253 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.150233    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 27 17:02:18.150920 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.150899    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 27 17:02:18.151035 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.151020    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:02:18.156833 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.156811    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g"]
Jul 27 17:02:18.234618 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.234588    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a26877e-9895-4df2-b890-bf52e79a3239-tmp\") pod \"insights-operator-5885fdb759-2vf8b\" (UID: \"9a26877e-9895-4df2-b890-bf52e79a3239\") " pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.234618 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.234622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9a26877e-9895-4df2-b890-bf52e79a3239-snapshots\") pod \"insights-operator-5885fdb759-2vf8b\" (UID: \"9a26877e-9895-4df2-b890-bf52e79a3239\") " pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.234860 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.234643    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4tlp\" (UniqueName: \"kubernetes.io/projected/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-kube-api-access-z4tlp\") pod \"cluster-samples-operator-6748489656-ftc4g\" (UID: \"9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g"
Jul 27 17:02:18.234860 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.234776    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a26877e-9895-4df2-b890-bf52e79a3239-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-2vf8b\" (UID: \"9a26877e-9895-4df2-b890-bf52e79a3239\") " pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.234860 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.234806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a26877e-9895-4df2-b890-bf52e79a3239-service-ca-bundle\") pod \"insights-operator-5885fdb759-2vf8b\" (UID: \"9a26877e-9895-4df2-b890-bf52e79a3239\") " pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.234860 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.234834    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-ftc4g\" (UID: \"9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g"
Jul 27 17:02:18.235028 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.234877    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a26877e-9895-4df2-b890-bf52e79a3239-serving-cert\") pod \"insights-operator-5885fdb759-2vf8b\" (UID: \"9a26877e-9895-4df2-b890-bf52e79a3239\") " pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.235028 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.234897    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/50804eac-c593-411c-9aaa-f4a3742cb1fe-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-dzpt4\" (UID: \"50804eac-c593-411c-9aaa-f4a3742cb1fe\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"
Jul 27 17:02:18.235028 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.234924    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ttxd\" (UniqueName: \"kubernetes.io/projected/50804eac-c593-411c-9aaa-f4a3742cb1fe-kube-api-access-7ttxd\") pod \"cluster-monitoring-operator-548b489cb8-dzpt4\" (UID: \"50804eac-c593-411c-9aaa-f4a3742cb1fe\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"
Jul 27 17:02:18.235028 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.234947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dzpt4\" (UID: \"50804eac-c593-411c-9aaa-f4a3742cb1fe\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"
Jul 27 17:02:18.235028 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.234972    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a26877e-9895-4df2-b890-bf52e79a3239-tmp\") pod \"insights-operator-5885fdb759-2vf8b\" (UID: \"9a26877e-9895-4df2-b890-bf52e79a3239\") " pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.235028 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.234983    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6p2r\" (UniqueName: \"kubernetes.io/projected/9a26877e-9895-4df2-b890-bf52e79a3239-kube-api-access-h6p2r\") pod \"insights-operator-5885fdb759-2vf8b\" (UID: \"9a26877e-9895-4df2-b890-bf52e79a3239\") " pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.235329 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:18.235086    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 17:02:18.235329 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:18.235155    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls podName:50804eac-c593-411c-9aaa-f4a3742cb1fe nodeName:}" failed. No retries permitted until 2026-07-27 17:02:18.735135253 +0000 UTC m=+82.810399735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-dzpt4" (UID: "50804eac-c593-411c-9aaa-f4a3742cb1fe") : secret "cluster-monitoring-operator-tls" not found
Jul 27 17:02:18.235458 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.235434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9a26877e-9895-4df2-b890-bf52e79a3239-snapshots\") pod \"insights-operator-5885fdb759-2vf8b\" (UID: \"9a26877e-9895-4df2-b890-bf52e79a3239\") " pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.235543 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.235513    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a26877e-9895-4df2-b890-bf52e79a3239-service-ca-bundle\") pod \"insights-operator-5885fdb759-2vf8b\" (UID: \"9a26877e-9895-4df2-b890-bf52e79a3239\") " pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.235687 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.235672    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a26877e-9895-4df2-b890-bf52e79a3239-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-2vf8b\" (UID: \"9a26877e-9895-4df2-b890-bf52e79a3239\") " pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.235912 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.235893    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/50804eac-c593-411c-9aaa-f4a3742cb1fe-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-dzpt4\" (UID: \"50804eac-c593-411c-9aaa-f4a3742cb1fe\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"
Jul 27 17:02:18.237181 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.237153    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a26877e-9895-4df2-b890-bf52e79a3239-serving-cert\") pod \"insights-operator-5885fdb759-2vf8b\" (UID: \"9a26877e-9895-4df2-b890-bf52e79a3239\") " pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.262468 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.262438    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ttxd\" (UniqueName: \"kubernetes.io/projected/50804eac-c593-411c-9aaa-f4a3742cb1fe-kube-api-access-7ttxd\") pod \"cluster-monitoring-operator-548b489cb8-dzpt4\" (UID: \"50804eac-c593-411c-9aaa-f4a3742cb1fe\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"
Jul 27 17:02:18.264740 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.264717    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6p2r\" (UniqueName: \"kubernetes.io/projected/9a26877e-9895-4df2-b890-bf52e79a3239-kube-api-access-h6p2r\") pod \"insights-operator-5885fdb759-2vf8b\" (UID: \"9a26877e-9895-4df2-b890-bf52e79a3239\") " pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.335807 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.335718    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4tlp\" (UniqueName: \"kubernetes.io/projected/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-kube-api-access-z4tlp\") pod \"cluster-samples-operator-6748489656-ftc4g\" (UID: \"9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g"
Jul 27 17:02:18.335807 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.335771    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-ftc4g\" (UID: \"9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g"
Jul 27 17:02:18.336004 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:18.335873    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 17:02:18.336004 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:18.335930    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-samples-operator-tls podName:9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:18.835916233 +0000 UTC m=+82.911180699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-samples-operator-tls") pod "cluster-samples-operator-6748489656-ftc4g" (UID: "9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60") : secret "samples-operator-tls" not found
Jul 27 17:02:18.348058 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.348034    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4tlp\" (UniqueName: \"kubernetes.io/projected/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-kube-api-access-z4tlp\") pod \"cluster-samples-operator-6748489656-ftc4g\" (UID: \"9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g"
Jul 27 17:02:18.363981 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.363959    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-2vf8b"
Jul 27 17:02:18.489544 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.489510    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-2vf8b"]
Jul 27 17:02:18.492554 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:02:18.492528    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a26877e_9895_4df2_b890_bf52e79a3239.slice/crio-c4da44cfc72a0d92d30aefdc3e923cc23be209af3d86b8805e856a7860cfb86e WatchSource:0}: Error finding container c4da44cfc72a0d92d30aefdc3e923cc23be209af3d86b8805e856a7860cfb86e: Status 404 returned error can't find the container with id c4da44cfc72a0d92d30aefdc3e923cc23be209af3d86b8805e856a7860cfb86e
Jul 27 17:02:18.728129 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.728086    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-2vf8b" event={"ID":"9a26877e-9895-4df2-b890-bf52e79a3239","Type":"ContainerStarted","Data":"c4da44cfc72a0d92d30aefdc3e923cc23be209af3d86b8805e856a7860cfb86e"}
Jul 27 17:02:18.739526 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.739502    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dzpt4\" (UID: \"50804eac-c593-411c-9aaa-f4a3742cb1fe\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"
Jul 27 17:02:18.739631 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:18.739619    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 17:02:18.739678 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:18.739670    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls podName:50804eac-c593-411c-9aaa-f4a3742cb1fe nodeName:}" failed. No retries permitted until 2026-07-27 17:02:19.739657244 +0000 UTC m=+83.814921708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-dzpt4" (UID: "50804eac-c593-411c-9aaa-f4a3742cb1fe") : secret "cluster-monitoring-operator-tls" not found
Jul 27 17:02:18.840525 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:18.840485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-ftc4g\" (UID: \"9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g"
Jul 27 17:02:18.840699 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:18.840638    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 17:02:18.840747 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:18.840708    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-samples-operator-tls podName:9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:19.840690911 +0000 UTC m=+83.915955382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-samples-operator-tls") pod "cluster-samples-operator-6748489656-ftc4g" (UID: "9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60") : secret "samples-operator-tls" not found
Jul 27 17:02:19.749414 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:19.749371    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dzpt4\" (UID: \"50804eac-c593-411c-9aaa-f4a3742cb1fe\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"
Jul 27 17:02:19.749885 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:19.749495    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 17:02:19.749885 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:19.749569    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls podName:50804eac-c593-411c-9aaa-f4a3742cb1fe nodeName:}" failed. No retries permitted until 2026-07-27 17:02:21.749547359 +0000 UTC m=+85.824811844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-dzpt4" (UID: "50804eac-c593-411c-9aaa-f4a3742cb1fe") : secret "cluster-monitoring-operator-tls" not found
Jul 27 17:02:19.850045 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:19.850005    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-ftc4g\" (UID: \"9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g"
Jul 27 17:02:19.850224 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:19.850202    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 17:02:19.850285 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:19.850274    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-samples-operator-tls podName:9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:21.850257333 +0000 UTC m=+85.925521798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-samples-operator-tls") pod "cluster-samples-operator-6748489656-ftc4g" (UID: "9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60") : secret "samples-operator-tls" not found
Jul 27 17:02:20.733677 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:20.733640    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-2vf8b" event={"ID":"9a26877e-9895-4df2-b890-bf52e79a3239","Type":"ContainerStarted","Data":"e3fa10fd3a271f1919b468f33bffe1cd4f17bbf842f26f75fd4c2bde885108a2"}
Jul 27 17:02:20.756595 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:20.756547    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-2vf8b" podStartSLOduration=0.910103291 podStartE2EDuration="2.756534034s" totalImagesPullingTime="1.846430743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:20.75616014 +0000 UTC m=+84.831424627" watchObservedRunningTime="2026-07-27 17:02:20.756534034 +0000 UTC m=+84.831798521"
Jul 27 17:02:21.764981 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:21.764930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dzpt4\" (UID: \"50804eac-c593-411c-9aaa-f4a3742cb1fe\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"
Jul 27 17:02:21.765486 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:21.765066    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 17:02:21.765486 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:21.765125    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls podName:50804eac-c593-411c-9aaa-f4a3742cb1fe nodeName:}" failed. No retries permitted until 2026-07-27 17:02:25.765110934 +0000 UTC m=+89.840375398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-dzpt4" (UID: "50804eac-c593-411c-9aaa-f4a3742cb1fe") : secret "cluster-monitoring-operator-tls" not found
Jul 27 17:02:21.865765 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:21.865704    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-ftc4g\" (UID: \"9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g"
Jul 27 17:02:21.865967 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:21.865871    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 17:02:21.865967 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:21.865964    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-samples-operator-tls podName:9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:25.86594205 +0000 UTC m=+89.941206527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-samples-operator-tls") pod "cluster-samples-operator-6748489656-ftc4g" (UID: "9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60") : secret "samples-operator-tls" not found
Jul 27 17:02:22.162585 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.162493    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l6rw6"]
Jul 27 17:02:22.166133 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.166108    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:22.169019 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.168994    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 17:02:22.169296 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.169265    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 17:02:22.169455 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.169302    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hn8cb\""
Jul 27 17:02:22.232226 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.232189    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l6rw6"]
Jul 27 17:02:22.267868 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.267836    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e299c5a3-e107-464a-ac41-444f951b507c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:22.268049 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.267892    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjl5q\" (UniqueName: \"kubernetes.io/projected/e299c5a3-e107-464a-ac41-444f951b507c-kube-api-access-wjl5q\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:22.268049 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.268015    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e299c5a3-e107-464a-ac41-444f951b507c-crio-socket\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:22.268143 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.268066    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e299c5a3-e107-464a-ac41-444f951b507c-data-volume\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:22.268143 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.268096    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:22.369288 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.369249    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e299c5a3-e107-464a-ac41-444f951b507c-data-volume\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:22.369288 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.369286    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:22.369485 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.369318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e299c5a3-e107-464a-ac41-444f951b507c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:22.369485 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.369357    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjl5q\" (UniqueName: \"kubernetes.io/projected/e299c5a3-e107-464a-ac41-444f951b507c-kube-api-access-wjl5q\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:22.369485 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:22.369415    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:22.369485 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.369431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e299c5a3-e107-464a-ac41-444f951b507c-crio-socket\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:22.369485 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:22.369450    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:02:22.369485 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:22.369475    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls podName:e299c5a3-e107-464a-ac41-444f951b507c nodeName:}" failed. No retries permitted until 2026-07-27 17:02:22.869460128 +0000 UTC m=+86.944724593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l6rw6" (UID: "e299c5a3-e107-464a-ac41-444f951b507c") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:22.369768 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.369508    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e299c5a3-e107-464a-ac41-444f951b507c-crio-socket\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:22.369768 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:22.369513    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e299c5a3-e107-464a-ac41-444f951b507c-kube-rbac-proxy-cm podName:e299c5a3-e107-464a-ac41-444f951b507c nodeName:}" failed. No retries permitted until 2026-07-27 17:02:22.869496832 +0000 UTC m=+86.944761320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e299c5a3-e107-464a-ac41-444f951b507c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l6rw6" (UID: "e299c5a3-e107-464a-ac41-444f951b507c") : configmap "kube-rbac-proxy" not found
Jul 27 17:02:22.369874 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.369827    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e299c5a3-e107-464a-ac41-444f951b507c-data-volume\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:22.380080 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.380053    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjl5q\" (UniqueName: \"kubernetes.io/projected/e299c5a3-e107-464a-ac41-444f951b507c-kube-api-access-wjl5q\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:22.874441 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.874396    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:22.874825 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:22.874469    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e299c5a3-e107-464a-ac41-444f951b507c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:22.874825 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:22.874566    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:22.874825 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:22.874599    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:02:22.874825 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:22.874628    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls podName:e299c5a3-e107-464a-ac41-444f951b507c nodeName:}" failed. No retries permitted until 2026-07-27 17:02:23.874612374 +0000 UTC m=+87.949876852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l6rw6" (UID: "e299c5a3-e107-464a-ac41-444f951b507c") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:22.874825 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:22.874652    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e299c5a3-e107-464a-ac41-444f951b507c-kube-rbac-proxy-cm podName:e299c5a3-e107-464a-ac41-444f951b507c nodeName:}" failed. No retries permitted until 2026-07-27 17:02:23.874639505 +0000 UTC m=+87.949903971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e299c5a3-e107-464a-ac41-444f951b507c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l6rw6" (UID: "e299c5a3-e107-464a-ac41-444f951b507c") : configmap "kube-rbac-proxy" not found
Jul 27 17:02:23.881743 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:23.881707    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:23.882111 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:23.881753    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e299c5a3-e107-464a-ac41-444f951b507c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:23.882111 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:23.881819    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:02:23.882111 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:23.881858    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:23.882111 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:23.881864    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e299c5a3-e107-464a-ac41-444f951b507c-kube-rbac-proxy-cm podName:e299c5a3-e107-464a-ac41-444f951b507c nodeName:}" failed. No retries permitted until 2026-07-27 17:02:25.88185142 +0000 UTC m=+89.957115886 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e299c5a3-e107-464a-ac41-444f951b507c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l6rw6" (UID: "e299c5a3-e107-464a-ac41-444f951b507c") : configmap "kube-rbac-proxy" not found
Jul 27 17:02:23.882111 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:23.881961    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls podName:e299c5a3-e107-464a-ac41-444f951b507c nodeName:}" failed. No retries permitted until 2026-07-27 17:02:25.881949075 +0000 UTC m=+89.957213543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l6rw6" (UID: "e299c5a3-e107-464a-ac41-444f951b507c") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:24.177277 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:24.177209    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6kcft_e02881b5-7256-40d5-9ad4-199b7c349a48/dns-node-resolver/0.log"
Jul 27 17:02:24.965520 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:24.965493    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2hn7j_676386a3-b937-45a3-88e9-54649e0839e0/node-ca/0.log"
Jul 27 17:02:25.761371 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:25.761338    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6tmv2_9d9ff7dd-7fa8-4a73-9b78-76fdd0531ef7/serve-healthcheck-canary/0.log"
Jul 27 17:02:25.798315 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:25.798276    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dzpt4\" (UID: \"50804eac-c593-411c-9aaa-f4a3742cb1fe\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"
Jul 27 17:02:25.798472 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:25.798429    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 17:02:25.798510 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:25.798491    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls podName:50804eac-c593-411c-9aaa-f4a3742cb1fe nodeName:}" failed. No retries permitted until 2026-07-27 17:02:33.798477817 +0000 UTC m=+97.873742282 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-dzpt4" (UID: "50804eac-c593-411c-9aaa-f4a3742cb1fe") : secret "cluster-monitoring-operator-tls" not found
Jul 27 17:02:25.899495 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:25.899457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e299c5a3-e107-464a-ac41-444f951b507c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:25.899595 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:25.899507    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-ftc4g\" (UID: \"9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g"
Jul 27 17:02:25.899595 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:25.899579    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:25.899688 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:25.899592    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:02:25.899688 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:25.899663    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e299c5a3-e107-464a-ac41-444f951b507c-kube-rbac-proxy-cm podName:e299c5a3-e107-464a-ac41-444f951b507c nodeName:}" failed. No retries permitted until 2026-07-27 17:02:29.899646333 +0000 UTC m=+93.974910802 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e299c5a3-e107-464a-ac41-444f951b507c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l6rw6" (UID: "e299c5a3-e107-464a-ac41-444f951b507c") : configmap "kube-rbac-proxy" not found
Jul 27 17:02:25.899758 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:25.899694    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 17:02:25.899758 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:25.899741    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-samples-operator-tls podName:9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:33.899729811 +0000 UTC m=+97.974994279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-samples-operator-tls") pod "cluster-samples-operator-6748489656-ftc4g" (UID: "9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60") : secret "samples-operator-tls" not found
Jul 27 17:02:25.899758 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:25.899694    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:25.899855 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:25.899772    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls podName:e299c5a3-e107-464a-ac41-444f951b507c nodeName:}" failed. No retries permitted until 2026-07-27 17:02:29.899763748 +0000 UTC m=+93.975028213 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l6rw6" (UID: "e299c5a3-e107-464a-ac41-444f951b507c") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:29.929734 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:29.929675    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:29.929734 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:29.929743    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e299c5a3-e107-464a-ac41-444f951b507c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:29.930340 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:29.929807    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:02:29.930340 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:29.929865    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:29.930340 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:29.930015    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e299c5a3-e107-464a-ac41-444f951b507c-kube-rbac-proxy-cm podName:e299c5a3-e107-464a-ac41-444f951b507c nodeName:}" failed. No retries permitted until 2026-07-27 17:02:37.929989186 +0000 UTC m=+102.005253655 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e299c5a3-e107-464a-ac41-444f951b507c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l6rw6" (UID: "e299c5a3-e107-464a-ac41-444f951b507c") : configmap "kube-rbac-proxy" not found
Jul 27 17:02:29.930340 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:29.930053    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls podName:e299c5a3-e107-464a-ac41-444f951b507c nodeName:}" failed. No retries permitted until 2026-07-27 17:02:37.930045606 +0000 UTC m=+102.005310071 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l6rw6" (UID: "e299c5a3-e107-464a-ac41-444f951b507c") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:33.860505 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:33.860469    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dzpt4\" (UID: \"50804eac-c593-411c-9aaa-f4a3742cb1fe\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"
Jul 27 17:02:33.860881 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:33.860624    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 17:02:33.860881 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:33.860711    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls podName:50804eac-c593-411c-9aaa-f4a3742cb1fe nodeName:}" failed. No retries permitted until 2026-07-27 17:02:49.860691055 +0000 UTC m=+113.935955521 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-dzpt4" (UID: "50804eac-c593-411c-9aaa-f4a3742cb1fe") : secret "cluster-monitoring-operator-tls" not found
Jul 27 17:02:33.961518 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:33.961476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-ftc4g\" (UID: \"9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g"
Jul 27 17:02:33.963800 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:33.963777    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-ftc4g\" (UID: \"9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g"
Jul 27 17:02:34.055932 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:34.055895    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g"
Jul 27 17:02:34.177565 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:34.177527    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g"]
Jul 27 17:02:34.667394 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:34.667345    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls\") pod \"dns-default-prbcj\" (UID: \"acde9127-43b5-46ed-9a95-515b25b0faf7\") " pod="openshift-dns/dns-default-prbcj"
Jul 27 17:02:34.667579 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:34.667422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:02:34.669686 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:34.669657    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acde9127-43b5-46ed-9a95-515b25b0faf7-metrics-tls\") pod \"dns-default-prbcj\" (UID: \"acde9127-43b5-46ed-9a95-515b25b0faf7\") " pod="openshift-dns/dns-default-prbcj"
Jul 27 17:02:34.669810 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:34.669781    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls\") pod \"image-registry-6cf44c948f-tc7r6\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") " pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:02:34.679978 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:34.679952    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5sbpw\""
Jul 27 17:02:34.688240 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:34.688220    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-prbcj"
Jul 27 17:02:34.768994 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:34.768966    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g" event={"ID":"9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60","Type":"ContainerStarted","Data":"23010311abeb82d18bd89a2ac1726b1d74d4a95cdea9a5b8afd3f5d15f16c578"}
Jul 27 17:02:34.807441 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:34.807405    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-prbcj"]
Jul 27 17:02:34.810312 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:02:34.810284    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacde9127_43b5_46ed_9a95_515b25b0faf7.slice/crio-4e797ac02bc2a0e70a70326c6b6d89205e6ea18a1dbb0ac0949921342bc621e5 WatchSource:0}: Error finding container 4e797ac02bc2a0e70a70326c6b6d89205e6ea18a1dbb0ac0949921342bc621e5: Status 404 returned error can't find the container with id 4e797ac02bc2a0e70a70326c6b6d89205e6ea18a1dbb0ac0949921342bc621e5
Jul 27 17:02:34.965677 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:34.965647    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-64scz\""
Jul 27 17:02:34.973070 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:34.973045    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:02:35.105091 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:35.105062    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cf44c948f-tc7r6"]
Jul 27 17:02:35.105335 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:02:35.105315    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79213db4_70fd_4b19_aaf6_d052590ae819.slice/crio-69d2fc0c8e5129c9bb74ad0a3d3e24fa554e3382542c622adbfca6e402a96e33 WatchSource:0}: Error finding container 69d2fc0c8e5129c9bb74ad0a3d3e24fa554e3382542c622adbfca6e402a96e33: Status 404 returned error can't find the container with id 69d2fc0c8e5129c9bb74ad0a3d3e24fa554e3382542c622adbfca6e402a96e33
Jul 27 17:02:35.774535 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:35.774482    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6" event={"ID":"79213db4-70fd-4b19-aaf6-d052590ae819","Type":"ContainerStarted","Data":"712bdd2211738a55d78a7a3fc407e6721a54f073e85e44d25ff18eebe1ff4200"}
Jul 27 17:02:35.774719 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:35.774612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6" event={"ID":"79213db4-70fd-4b19-aaf6-d052590ae819","Type":"ContainerStarted","Data":"69d2fc0c8e5129c9bb74ad0a3d3e24fa554e3382542c622adbfca6e402a96e33"}
Jul 27 17:02:35.774851 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:35.774823    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:02:35.775950 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:35.775871    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-prbcj" event={"ID":"acde9127-43b5-46ed-9a95-515b25b0faf7","Type":"ContainerStarted","Data":"4e797ac02bc2a0e70a70326c6b6d89205e6ea18a1dbb0ac0949921342bc621e5"}
Jul 27 17:02:35.796186 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:35.796117    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6" podStartSLOduration=98.796097721 podStartE2EDuration="1m38.796097721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:35.794229299 +0000 UTC m=+99.869493787" watchObservedRunningTime="2026-07-27 17:02:35.796097721 +0000 UTC m=+99.871362207"
Jul 27 17:02:37.784370 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:37.784326    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g" event={"ID":"9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60","Type":"ContainerStarted","Data":"06c731f559c0aaeed7749448c01668450b10edbbf362fad7405921c4d86b10dc"}
Jul 27 17:02:37.786578 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:37.786553    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-prbcj" event={"ID":"acde9127-43b5-46ed-9a95-515b25b0faf7","Type":"ContainerStarted","Data":"9caa1fc3fe4b5977a1b9e72be737424f6ceccde43ff86a4dbfefe98948de0b2a"}
Jul 27 17:02:37.996065 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:37.996028    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e299c5a3-e107-464a-ac41-444f951b507c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:37.996266 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:37.996120    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:37.996266 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:37.996239    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:37.996339 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:37.996293    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls podName:e299c5a3-e107-464a-ac41-444f951b507c nodeName:}" failed. No retries permitted until 2026-07-27 17:02:53.996279382 +0000 UTC m=+118.071543847 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l6rw6" (UID: "e299c5a3-e107-464a-ac41-444f951b507c") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:37.996606 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:37.996583    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e299c5a3-e107-464a-ac41-444f951b507c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:38.703944 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:38.703916    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6pt78"
Jul 27 17:02:38.791928 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:38.791890    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g" event={"ID":"9d8c8ab5-0263-4c6a-82c1-ac3ba6c10a60","Type":"ContainerStarted","Data":"70bedc7fd92e516ae15e76b21c474472f62bcb995d46fcaa6633323be173e271"}
Jul 27 17:02:38.793566 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:38.793543    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-prbcj" event={"ID":"acde9127-43b5-46ed-9a95-515b25b0faf7","Type":"ContainerStarted","Data":"3db512bdd9d1d4b8538f079bff0c71bacb154b953daba069218d647f019e59e8"}
Jul 27 17:02:38.793674 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:38.793600    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-prbcj"
Jul 27 17:02:38.808922 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:38.808879    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ftc4g" podStartSLOduration=17.513594105 podStartE2EDuration="20.808863892s" totalImagesPullingTime="3.295269787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:38.80835911 +0000 UTC m=+102.883623598" watchObservedRunningTime="2026-07-27 17:02:38.808863892 +0000 UTC m=+102.884128378"
Jul 27 17:02:38.826742 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:38.826690    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-prbcj" podStartSLOduration=66.123543656 podStartE2EDuration="1m8.826674065s" totalImagesPullingTime="2.703130409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:38.825633991 +0000 UTC m=+102.900898477" watchObservedRunningTime="2026-07-27 17:02:38.826674065 +0000 UTC m=+102.901938553"
Jul 27 17:02:47.499512 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:47.499475    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79549c744b-srv6c"]
Jul 27 17:02:47.502378 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:47.502362    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79549c744b-srv6c"
Jul 27 17:02:47.505865 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:47.505834    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 27 17:02:47.505865 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:47.505857    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 17:02:47.506065 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:47.505958    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 17:02:47.506126 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:47.506101    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-cx922\""
Jul 27 17:02:47.506204 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:47.506138    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 17:02:47.521434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:47.521409    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79549c744b-srv6c"]
Jul 27 17:02:47.566381 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:47.566351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcg4r\" (UniqueName: \"kubernetes.io/projected/2148f94d-c4da-4e76-919b-498d702ae6f1-kube-api-access-fcg4r\") pod \"managed-serviceaccount-addon-agent-79549c744b-srv6c\" (UID: \"2148f94d-c4da-4e76-919b-498d702ae6f1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79549c744b-srv6c"
Jul 27 17:02:47.566539 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:47.566461    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2148f94d-c4da-4e76-919b-498d702ae6f1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79549c744b-srv6c\" (UID: \"2148f94d-c4da-4e76-919b-498d702ae6f1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79549c744b-srv6c"
Jul 27 17:02:47.667891 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:47.667852    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2148f94d-c4da-4e76-919b-498d702ae6f1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79549c744b-srv6c\" (UID: \"2148f94d-c4da-4e76-919b-498d702ae6f1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79549c744b-srv6c"
Jul 27 17:02:47.668059 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:47.667916    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcg4r\" (UniqueName: \"kubernetes.io/projected/2148f94d-c4da-4e76-919b-498d702ae6f1-kube-api-access-fcg4r\") pod \"managed-serviceaccount-addon-agent-79549c744b-srv6c\" (UID: \"2148f94d-c4da-4e76-919b-498d702ae6f1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79549c744b-srv6c"
Jul 27 17:02:47.670935 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:47.670909    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2148f94d-c4da-4e76-919b-498d702ae6f1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79549c744b-srv6c\" (UID: \"2148f94d-c4da-4e76-919b-498d702ae6f1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79549c744b-srv6c"
Jul 27 17:02:47.681514 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:47.681485    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcg4r\" (UniqueName: \"kubernetes.io/projected/2148f94d-c4da-4e76-919b-498d702ae6f1-kube-api-access-fcg4r\") pod \"managed-serviceaccount-addon-agent-79549c744b-srv6c\" (UID: \"2148f94d-c4da-4e76-919b-498d702ae6f1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79549c744b-srv6c"
Jul 27 17:02:47.831408 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:47.831323    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79549c744b-srv6c"
Jul 27 17:02:47.969730 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:47.969690    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79549c744b-srv6c"]
Jul 27 17:02:47.972917 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:02:47.972889    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2148f94d_c4da_4e76_919b_498d702ae6f1.slice/crio-36f6e1ffbd7a49613eff7f92c41049edc59985ed47a2de8e5a703af09af230de WatchSource:0}: Error finding container 36f6e1ffbd7a49613eff7f92c41049edc59985ed47a2de8e5a703af09af230de: Status 404 returned error can't find the container with id 36f6e1ffbd7a49613eff7f92c41049edc59985ed47a2de8e5a703af09af230de
Jul 27 17:02:48.799257 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:48.799224    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-prbcj"
Jul 27 17:02:48.828068 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:48.828035    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79549c744b-srv6c" event={"ID":"2148f94d-c4da-4e76-919b-498d702ae6f1","Type":"ContainerStarted","Data":"36f6e1ffbd7a49613eff7f92c41049edc59985ed47a2de8e5a703af09af230de"}
Jul 27 17:02:49.884424 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:49.884389    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dzpt4\" (UID: \"50804eac-c593-411c-9aaa-f4a3742cb1fe\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"
Jul 27 17:02:49.886808 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:49.886773    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/50804eac-c593-411c-9aaa-f4a3742cb1fe-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-dzpt4\" (UID: \"50804eac-c593-411c-9aaa-f4a3742cb1fe\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"
Jul 27 17:02:50.157297 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:50.157210    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"
Jul 27 17:02:51.108095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:51.108065    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4"]
Jul 27 17:02:51.111447 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:02:51.111422    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50804eac_c593_411c_9aaa_f4a3742cb1fe.slice/crio-770b47730f671bb688e175c72818eca33c07ad172f6767487a603cc2752e845c WatchSource:0}: Error finding container 770b47730f671bb688e175c72818eca33c07ad172f6767487a603cc2752e845c: Status 404 returned error can't find the container with id 770b47730f671bb688e175c72818eca33c07ad172f6767487a603cc2752e845c
Jul 27 17:02:51.840697 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:51.840660    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79549c744b-srv6c" event={"ID":"2148f94d-c4da-4e76-919b-498d702ae6f1","Type":"ContainerStarted","Data":"2dcbbec0460c654a83ef547a3399d7cc2fdda67324e4e0299025271b9cae92b1"}
Jul 27 17:02:51.841747 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:51.841719    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4" event={"ID":"50804eac-c593-411c-9aaa-f4a3742cb1fe","Type":"ContainerStarted","Data":"770b47730f671bb688e175c72818eca33c07ad172f6767487a603cc2752e845c"}
Jul 27 17:02:51.855425 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:51.855306    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79549c744b-srv6c" podStartSLOduration=1.780268993 podStartE2EDuration="4.855289229s" totalImagesPullingTime="3.075020236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:51.855281603 +0000 UTC m=+115.930546093" watchObservedRunningTime="2026-07-27 17:02:51.855289229 +0000 UTC m=+115.930553718"
Jul 27 17:02:53.482085 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:53.482049    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g"]
Jul 27 17:02:53.485067 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:53.485051    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g"
Jul 27 17:02:53.487591 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:53.487566    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 27 17:02:53.487692 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:53.487621    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kp2tf\""
Jul 27 17:02:53.493365 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:53.493344    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g"]
Jul 27 17:02:53.612931 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:53.612894    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b9850d85-2dad-44dc-83a3-4edf23d796a5-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-mgf8g\" (UID: \"b9850d85-2dad-44dc-83a3-4edf23d796a5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g"
Jul 27 17:02:53.713568 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:53.713530    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b9850d85-2dad-44dc-83a3-4edf23d796a5-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-mgf8g\" (UID: \"b9850d85-2dad-44dc-83a3-4edf23d796a5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g"
Jul 27 17:02:53.713722 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:53.713684    2566 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 27 17:02:53.713762 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:53.713752    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9850d85-2dad-44dc-83a3-4edf23d796a5-tls-certificates podName:b9850d85-2dad-44dc-83a3-4edf23d796a5 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:54.213736429 +0000 UTC m=+118.289000893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/b9850d85-2dad-44dc-83a3-4edf23d796a5-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-mgf8g" (UID: "b9850d85-2dad-44dc-83a3-4edf23d796a5") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 27 17:02:53.849764 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:53.849693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4" event={"ID":"50804eac-c593-411c-9aaa-f4a3742cb1fe","Type":"ContainerStarted","Data":"217d07870ae2bb2c5b39eeac5c4c3c644ced91483ae7059381b2900977bd7de2"}
Jul 27 17:02:53.867844 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:53.867797    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-dzpt4" podStartSLOduration=33.980453823 podStartE2EDuration="35.867782535s" totalImagesPullingTime="1.887328712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:53.866474646 +0000 UTC m=+117.941739132" watchObservedRunningTime="2026-07-27 17:02:53.867782535 +0000 UTC m=+117.943047021"
Jul 27 17:02:54.016470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:54.016431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:54.019404 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:54.019376    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e299c5a3-e107-464a-ac41-444f951b507c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l6rw6\" (UID: \"e299c5a3-e107-464a-ac41-444f951b507c\") " pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:54.218295 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:54.218259    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b9850d85-2dad-44dc-83a3-4edf23d796a5-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-mgf8g\" (UID: \"b9850d85-2dad-44dc-83a3-4edf23d796a5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g"
Jul 27 17:02:54.220511 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:54.220493    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b9850d85-2dad-44dc-83a3-4edf23d796a5-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-mgf8g\" (UID: \"b9850d85-2dad-44dc-83a3-4edf23d796a5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g"
Jul 27 17:02:54.276643 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:54.276620    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l6rw6"
Jul 27 17:02:54.393447 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:54.393414    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g"
Jul 27 17:02:54.405283 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:54.405260    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l6rw6"]
Jul 27 17:02:54.407233 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:02:54.407203    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode299c5a3_e107_464a_ac41_444f951b507c.slice/crio-8a8a24cb48d6608c126d6ac803ebd3ce245df3155ced1105d72f8e5966e89a3a WatchSource:0}: Error finding container 8a8a24cb48d6608c126d6ac803ebd3ce245df3155ced1105d72f8e5966e89a3a: Status 404 returned error can't find the container with id 8a8a24cb48d6608c126d6ac803ebd3ce245df3155ced1105d72f8e5966e89a3a
Jul 27 17:02:54.513357 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:54.513328    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g"]
Jul 27 17:02:54.515806 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:02:54.515778    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9850d85_2dad_44dc_83a3_4edf23d796a5.slice/crio-654ebb6ddac8bef0f02b481fd74453706092df7c7cc44e846a80d6bd3cd3d622 WatchSource:0}: Error finding container 654ebb6ddac8bef0f02b481fd74453706092df7c7cc44e846a80d6bd3cd3d622: Status 404 returned error can't find the container with id 654ebb6ddac8bef0f02b481fd74453706092df7c7cc44e846a80d6bd3cd3d622
Jul 27 17:02:54.856336 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:54.856253    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g" event={"ID":"b9850d85-2dad-44dc-83a3-4edf23d796a5","Type":"ContainerStarted","Data":"654ebb6ddac8bef0f02b481fd74453706092df7c7cc44e846a80d6bd3cd3d622"}
Jul 27 17:02:54.860670 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:54.860641    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l6rw6" event={"ID":"e299c5a3-e107-464a-ac41-444f951b507c","Type":"ContainerStarted","Data":"8a8a24cb48d6608c126d6ac803ebd3ce245df3155ced1105d72f8e5966e89a3a"}
Jul 27 17:02:54.980286 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:54.980243    2566 patch_prober.go:28] interesting pod/image-registry-6cf44c948f-tc7r6 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 17:02:54.980460 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:54.980309    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6" podUID="79213db4-70fd-4b19-aaf6-d052590ae819" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:02:56.783981 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:56.783953    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:02:56.868726 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:56.868686    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g" event={"ID":"b9850d85-2dad-44dc-83a3-4edf23d796a5","Type":"ContainerStarted","Data":"48b74045324b2a1a044df35a57374c3d348e3f7c67abe871f01543d3962fd09a"}
Jul 27 17:02:56.868980 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:56.868956    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g"
Jul 27 17:02:56.875258 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:56.875233    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g"
Jul 27 17:02:56.886371 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:56.886312    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mgf8g" podStartSLOduration=2.594804222 podStartE2EDuration="3.886289403s" totalImagesPullingTime="1.291485181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:56.885370009 +0000 UTC m=+120.960634501" watchObservedRunningTime="2026-07-27 17:02:56.886289403 +0000 UTC m=+120.961553888"
Jul 27 17:02:57.546838 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:57.546802    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-p4prc"]
Jul 27 17:02:57.563480 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:57.563451    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-p4prc"]
Jul 27 17:02:57.563646 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:57.563567    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-p4prc"
Jul 27 17:02:57.567552 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:57.567303    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 27 17:02:57.567552 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:57.567328    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 17:02:57.567552 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:57.567307    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 27 17:02:57.567552 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:57.567307    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rcww2\""
Jul 27 17:02:57.647030 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:57.646996    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d663fddb-ec5a-48ed-907b-ab957759a948-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-p4prc\" (UID: \"d663fddb-ec5a-48ed-907b-ab957759a948\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-p4prc"
Jul 27 17:02:57.647222 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:57.647046    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5spx8\" (UniqueName: \"kubernetes.io/projected/d663fddb-ec5a-48ed-907b-ab957759a948-kube-api-access-5spx8\") pod \"prometheus-operator-84d69f794d-p4prc\" (UID: \"d663fddb-ec5a-48ed-907b-ab957759a948\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-p4prc"
Jul 27 17:02:57.647222 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:57.647182    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d663fddb-ec5a-48ed-907b-ab957759a948-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-p4prc\" (UID: \"d663fddb-ec5a-48ed-907b-ab957759a948\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-p4prc"
Jul 27 17:02:57.647335 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:57.647234    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d663fddb-ec5a-48ed-907b-ab957759a948-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-p4prc\" (UID: \"d663fddb-ec5a-48ed-907b-ab957759a948\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-p4prc"
Jul 27 17:02:57.748358 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:57.748321    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d663fddb-ec5a-48ed-907b-ab957759a948-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-p4prc\" (UID: \"d663fddb-ec5a-48ed-907b-ab957759a948\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-p4prc"
Jul 27 17:02:57.748525 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:57.748366    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d663fddb-ec5a-48ed-907b-ab957759a948-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-p4prc\" (UID: \"d663fddb-ec5a-48ed-907b-ab957759a948\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-p4prc"
Jul 27 17:02:57.748525 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:57.748403    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d663fddb-ec5a-48ed-907b-ab957759a948-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-p4prc\" (UID: \"d663fddb-ec5a-48ed-907b-ab957759a948\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-p4prc"
Jul 27 17:02:57.748525 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:57.748445    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5spx8\" (UniqueName: \"kubernetes.io/projected/d663fddb-ec5a-48ed-907b-ab957759a948-kube-api-access-5spx8\") pod \"prometheus-operator-84d69f794d-p4prc\" (UID: \"d663fddb-ec5a-48ed-907b-ab957759a948\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-p4prc"
Jul 27 17:02:57.748693 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:57.748565    2566 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 27 17:02:57.748693 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:02:57.748648    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d663fddb-ec5a-48ed-907b-ab957759a948-prometheus-operator-tls podName:d663fddb-ec5a-48ed-907b-ab957759a948 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:58.248628001 +0000 UTC m=+122.323892466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/d663fddb-ec5a-48ed-907b-ab957759a948-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-p4prc" (UID: "d663fddb-ec5a-48ed-907b-ab957759a948") : secret "prometheus-operator-tls" not found
Jul 27 17:02:57.749134 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:57.749110    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d663fddb-ec5a-48ed-907b-ab957759a948-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-p4prc\" (UID: \"d663fddb-ec5a-48ed-907b-ab957759a948\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-p4prc"
Jul 27 17:02:57.751097 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:57.751076    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d663fddb-ec5a-48ed-907b-ab957759a948-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-p4prc\" (UID: \"d663fddb-ec5a-48ed-907b-ab957759a948\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-p4prc"
Jul 27 17:02:57.759911 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:57.759892    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5spx8\" (UniqueName: \"kubernetes.io/projected/d663fddb-ec5a-48ed-907b-ab957759a948-kube-api-access-5spx8\") pod \"prometheus-operator-84d69f794d-p4prc\" (UID: \"d663fddb-ec5a-48ed-907b-ab957759a948\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-p4prc"
Jul 27 17:02:58.254155 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:58.254118    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d663fddb-ec5a-48ed-907b-ab957759a948-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-p4prc\" (UID: \"d663fddb-ec5a-48ed-907b-ab957759a948\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-p4prc"
Jul 27 17:02:58.256782 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:58.256738    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d663fddb-ec5a-48ed-907b-ab957759a948-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-p4prc\" (UID: \"d663fddb-ec5a-48ed-907b-ab957759a948\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-p4prc"
Jul 27 17:02:58.474971 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:58.474926    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-p4prc"
Jul 27 17:02:59.363286 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:59.363226    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-p4prc"]
Jul 27 17:02:59.365753 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:02:59.365730    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd663fddb_ec5a_48ed_907b_ab957759a948.slice/crio-88309bca0053aa80318afb6d7664fd433182e2b84eab20a94e4aacae64d8923c WatchSource:0}: Error finding container 88309bca0053aa80318afb6d7664fd433182e2b84eab20a94e4aacae64d8923c: Status 404 returned error can't find the container with id 88309bca0053aa80318afb6d7664fd433182e2b84eab20a94e4aacae64d8923c
Jul 27 17:02:59.883118 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:59.883077    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l6rw6" event={"ID":"e299c5a3-e107-464a-ac41-444f951b507c","Type":"ContainerStarted","Data":"b20e83789b6f8ded17f9a1f62a576fce10edcd10bd9fa5d90e030ce53c60e07f"}
Jul 27 17:02:59.884152 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:02:59.884129    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-p4prc" event={"ID":"d663fddb-ec5a-48ed-907b-ab957759a948","Type":"ContainerStarted","Data":"88309bca0053aa80318afb6d7664fd433182e2b84eab20a94e4aacae64d8923c"}
Jul 27 17:03:03.899991 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:03.899945    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-p4prc" event={"ID":"d663fddb-ec5a-48ed-907b-ab957759a948","Type":"ContainerStarted","Data":"78c2244f577d02b1b370204b0b0f62ad13b5abb3dfc290fc9d6f20700ac93294"}
Jul 27 17:03:03.900492 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:03.900024    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-p4prc" event={"ID":"d663fddb-ec5a-48ed-907b-ab957759a948","Type":"ContainerStarted","Data":"e75508cb6e8416f14557631e35b04dd70a19f403fbf6e94b8b28c1e94248960f"}
Jul 27 17:03:03.901767 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:03.901743    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l6rw6" event={"ID":"e299c5a3-e107-464a-ac41-444f951b507c","Type":"ContainerStarted","Data":"60b925960acb090baae78e6c588635e3c5fad488f40a1e5e5a002a1b7c2e516f"}
Jul 27 17:03:03.918430 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:03.918381    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-p4prc" podStartSLOduration=3.197474128 podStartE2EDuration="6.918364305s" totalImagesPullingTime="3.720890177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:03.916731526 +0000 UTC m=+127.991996012" watchObservedRunningTime="2026-07-27 17:03:03.918364305 +0000 UTC m=+127.993628793"
Jul 27 17:03:05.900718 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:05.900680    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6"]
Jul 27 17:03:05.916762 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:05.916727    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6"
Jul 27 17:03:05.919092 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:05.919063    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 27 17:03:05.919735 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:05.919578    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-q9mwd\""
Jul 27 17:03:05.919735 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:05.919624    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 27 17:03:05.919906 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:05.919883    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff30bbdb-a210-458d-b79e-be33cd423407-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-dgxj6\" (UID: \"ff30bbdb-a210-458d-b79e-be33cd423407\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6"
Jul 27 17:03:05.919969 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:05.919955    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t9rc\" (UniqueName: \"kubernetes.io/projected/ff30bbdb-a210-458d-b79e-be33cd423407-kube-api-access-6t9rc\") pod \"openshift-state-metrics-b8f49f5c6-dgxj6\" (UID: \"ff30bbdb-a210-458d-b79e-be33cd423407\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6"
Jul 27 17:03:05.920550 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:05.920022    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff30bbdb-a210-458d-b79e-be33cd423407-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-dgxj6\" (UID: \"ff30bbdb-a210-458d-b79e-be33cd423407\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6"
Jul 27 17:03:05.920550 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:05.920071    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff30bbdb-a210-458d-b79e-be33cd423407-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-dgxj6\" (UID: \"ff30bbdb-a210-458d-b79e-be33cd423407\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6"
Jul 27 17:03:05.921758 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:05.921740    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6"]
Jul 27 17:03:05.928841 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:05.928816    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7m5zw"]
Jul 27 17:03:05.933838 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:05.933818    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:05.937038 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:05.936986    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 17:03:05.937038 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:05.937018    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 17:03:05.937226 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:05.936993    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m4bj4\""
Jul 27 17:03:05.937226 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:05.937138    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 17:03:06.021239 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.021203    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48f4573e-95be-44cb-b1e4-6773d04d7f68-root\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.021239 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.021240    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48f4573e-95be-44cb-b1e4-6773d04d7f68-metrics-client-ca\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.021439 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.021265    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkszq\" (UniqueName: \"kubernetes.io/projected/48f4573e-95be-44cb-b1e4-6773d04d7f68-kube-api-access-lkszq\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.021439 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.021310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t9rc\" (UniqueName: \"kubernetes.io/projected/ff30bbdb-a210-458d-b79e-be33cd423407-kube-api-access-6t9rc\") pod \"openshift-state-metrics-b8f49f5c6-dgxj6\" (UID: \"ff30bbdb-a210-458d-b79e-be33cd423407\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6"
Jul 27 17:03:06.021439 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.021378    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48f4573e-95be-44cb-b1e4-6773d04d7f68-node-exporter-accelerators-collector-config\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.021439 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.021408    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48f4573e-95be-44cb-b1e4-6773d04d7f68-node-exporter-wtmp\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.021585 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.021493    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff30bbdb-a210-458d-b79e-be33cd423407-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-dgxj6\" (UID: \"ff30bbdb-a210-458d-b79e-be33cd423407\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6"
Jul 27 17:03:06.021585 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.021539    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff30bbdb-a210-458d-b79e-be33cd423407-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-dgxj6\" (UID: \"ff30bbdb-a210-458d-b79e-be33cd423407\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6"
Jul 27 17:03:06.021585 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.021571    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48f4573e-95be-44cb-b1e4-6773d04d7f68-node-exporter-textfile\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.021712 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.021596    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48f4573e-95be-44cb-b1e4-6773d04d7f68-node-exporter-tls\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.021712 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.021685    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48f4573e-95be-44cb-b1e4-6773d04d7f68-sys\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.021780 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.021723    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff30bbdb-a210-458d-b79e-be33cd423407-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-dgxj6\" (UID: \"ff30bbdb-a210-458d-b79e-be33cd423407\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6"
Jul 27 17:03:06.021780 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.021751    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48f4573e-95be-44cb-b1e4-6773d04d7f68-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.022398 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.022370    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff30bbdb-a210-458d-b79e-be33cd423407-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-dgxj6\" (UID: \"ff30bbdb-a210-458d-b79e-be33cd423407\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6"
Jul 27 17:03:06.024238 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.024218    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff30bbdb-a210-458d-b79e-be33cd423407-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-dgxj6\" (UID: \"ff30bbdb-a210-458d-b79e-be33cd423407\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6"
Jul 27 17:03:06.024533 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.024512    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff30bbdb-a210-458d-b79e-be33cd423407-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-dgxj6\" (UID: \"ff30bbdb-a210-458d-b79e-be33cd423407\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6"
Jul 27 17:03:06.042676 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.042650    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t9rc\" (UniqueName: \"kubernetes.io/projected/ff30bbdb-a210-458d-b79e-be33cd423407-kube-api-access-6t9rc\") pod \"openshift-state-metrics-b8f49f5c6-dgxj6\" (UID: \"ff30bbdb-a210-458d-b79e-be33cd423407\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6"
Jul 27 17:03:06.122934 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.122896    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48f4573e-95be-44cb-b1e4-6773d04d7f68-node-exporter-wtmp\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.123094 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.122945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48f4573e-95be-44cb-b1e4-6773d04d7f68-node-exporter-textfile\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.123094 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.123045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48f4573e-95be-44cb-b1e4-6773d04d7f68-node-exporter-tls\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.123240 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.123099    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48f4573e-95be-44cb-b1e4-6773d04d7f68-node-exporter-wtmp\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.123240 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.123122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48f4573e-95be-44cb-b1e4-6773d04d7f68-sys\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.123240 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.123154    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48f4573e-95be-44cb-b1e4-6773d04d7f68-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.123240 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.123207    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48f4573e-95be-44cb-b1e4-6773d04d7f68-root\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.123240 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:03:06.123224    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 17:03:06.123240 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.123232    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48f4573e-95be-44cb-b1e4-6773d04d7f68-metrics-client-ca\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.123533 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.123264    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkszq\" (UniqueName: \"kubernetes.io/projected/48f4573e-95be-44cb-b1e4-6773d04d7f68-kube-api-access-lkszq\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.123533 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:03:06.123283    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48f4573e-95be-44cb-b1e4-6773d04d7f68-node-exporter-tls podName:48f4573e-95be-44cb-b1e4-6773d04d7f68 nodeName:}" failed. No retries permitted until 2026-07-27 17:03:06.623262435 +0000 UTC m=+130.698526904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/48f4573e-95be-44cb-b1e4-6773d04d7f68-node-exporter-tls") pod "node-exporter-7m5zw" (UID: "48f4573e-95be-44cb-b1e4-6773d04d7f68") : secret "node-exporter-tls" not found
Jul 27 17:03:06.123533 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.123289    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48f4573e-95be-44cb-b1e4-6773d04d7f68-node-exporter-textfile\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.123533 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.123379    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48f4573e-95be-44cb-b1e4-6773d04d7f68-node-exporter-accelerators-collector-config\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.123731 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.123561    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48f4573e-95be-44cb-b1e4-6773d04d7f68-root\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.123907 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.123866    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48f4573e-95be-44cb-b1e4-6773d04d7f68-sys\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.124031 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.123956    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48f4573e-95be-44cb-b1e4-6773d04d7f68-node-exporter-accelerators-collector-config\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.124089 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.124043    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48f4573e-95be-44cb-b1e4-6773d04d7f68-metrics-client-ca\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.125837 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.125819    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48f4573e-95be-44cb-b1e4-6773d04d7f68-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.134717 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.134698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkszq\" (UniqueName: \"kubernetes.io/projected/48f4573e-95be-44cb-b1e4-6773d04d7f68-kube-api-access-lkszq\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.228858 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.228828    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6"
Jul 27 17:03:06.324597 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.324561    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs\") pod \"network-metrics-daemon-d72jm\" (UID: \"e0651122-b4fd-4a56-8909-53661d776bfc\") " pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:03:06.327153 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.327120    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0651122-b4fd-4a56-8909-53661d776bfc-metrics-certs\") pod \"network-metrics-daemon-d72jm\" (UID: \"e0651122-b4fd-4a56-8909-53661d776bfc\") " pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:03:06.512593 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.512520    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mxtqx\""
Jul 27 17:03:06.519902 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.519866    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d72jm"
Jul 27 17:03:06.626767 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.626731    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48f4573e-95be-44cb-b1e4-6773d04d7f68-node-exporter-tls\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.629156 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.629126    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48f4573e-95be-44cb-b1e4-6773d04d7f68-node-exporter-tls\") pod \"node-exporter-7m5zw\" (UID: \"48f4573e-95be-44cb-b1e4-6773d04d7f68\") " pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.796923 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.796897    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d72jm"]
Jul 27 17:03:06.800079 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:03:06.800053    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0651122_b4fd_4a56_8909_53661d776bfc.slice/crio-05fcbb48c3678063ce5e0c35159ac1f83576476a03aa3e829e87604b1b7c17f9 WatchSource:0}: Error finding container 05fcbb48c3678063ce5e0c35159ac1f83576476a03aa3e829e87604b1b7c17f9: Status 404 returned error can't find the container with id 05fcbb48c3678063ce5e0c35159ac1f83576476a03aa3e829e87604b1b7c17f9
Jul 27 17:03:06.811469 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.811443    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6"]
Jul 27 17:03:06.813465 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:03:06.813437    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff30bbdb_a210_458d_b79e_be33cd423407.slice/crio-c992db7aec4968218df492cc15bb7031015a54719fea575a66afe75a4894ccd1 WatchSource:0}: Error finding container c992db7aec4968218df492cc15bb7031015a54719fea575a66afe75a4894ccd1: Status 404 returned error can't find the container with id c992db7aec4968218df492cc15bb7031015a54719fea575a66afe75a4894ccd1
Jul 27 17:03:06.844018 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.843955    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7m5zw"
Jul 27 17:03:06.855037 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:03:06.855015    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f4573e_95be_44cb_b1e4_6773d04d7f68.slice/crio-f8be1b466805159038ac342fc44b6e55b2f35d4341fa45717d0d731903e3ecdb WatchSource:0}: Error finding container f8be1b466805159038ac342fc44b6e55b2f35d4341fa45717d0d731903e3ecdb: Status 404 returned error can't find the container with id f8be1b466805159038ac342fc44b6e55b2f35d4341fa45717d0d731903e3ecdb
Jul 27 17:03:06.915960 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.915869    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l6rw6" event={"ID":"e299c5a3-e107-464a-ac41-444f951b507c","Type":"ContainerStarted","Data":"0e80c55b19bc3e255d17cdf4e7416329bd217937c227c242c41b66299a73e95a"}
Jul 27 17:03:06.918061 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.918030    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7m5zw" event={"ID":"48f4573e-95be-44cb-b1e4-6773d04d7f68","Type":"ContainerStarted","Data":"f8be1b466805159038ac342fc44b6e55b2f35d4341fa45717d0d731903e3ecdb"}
Jul 27 17:03:06.919328 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.919299    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d72jm" event={"ID":"e0651122-b4fd-4a56-8909-53661d776bfc","Type":"ContainerStarted","Data":"05fcbb48c3678063ce5e0c35159ac1f83576476a03aa3e829e87604b1b7c17f9"}
Jul 27 17:03:06.920905 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.920879    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6" event={"ID":"ff30bbdb-a210-458d-b79e-be33cd423407","Type":"ContainerStarted","Data":"3db466dbe06013af32b1c1be41cbd6cefc40429c24a79d9706424af71e33b08f"}
Jul 27 17:03:06.921014 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.920923    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6" event={"ID":"ff30bbdb-a210-458d-b79e-be33cd423407","Type":"ContainerStarted","Data":"c992db7aec4968218df492cc15bb7031015a54719fea575a66afe75a4894ccd1"}
Jul 27 17:03:06.943298 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:06.943248    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l6rw6" podStartSLOduration=32.786252771 podStartE2EDuration="44.943161088s" totalImagesPullingTime="12.156908317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:02:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:06.941738538 +0000 UTC m=+131.017003026" watchObservedRunningTime="2026-07-27 17:03:06.943161088 +0000 UTC m=+131.018425610"
Jul 27 17:03:07.073461 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.073378    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:03:07.077133 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.077114    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.079579 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.079558    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 17:03:07.079701 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.079586    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 17:03:07.079701 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.079584    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 17:03:07.079817 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.079743    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 17:03:07.079904 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.079881    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 17:03:07.080018 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.079924    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 17:03:07.080152 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.080133    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 17:03:07.080251 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.080198    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 17:03:07.080251 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.080225    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m77t2\""
Jul 27 17:03:07.080365 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.080258    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 17:03:07.095483 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.095462    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:03:07.130111 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.130079    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a569873e-386e-4abd-8a51-3877414b204f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.130303 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.130177    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a569873e-386e-4abd-8a51-3877414b204f-config-out\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.130303 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.130246    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.130407 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.130304    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.130407 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.130364    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-web-config\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.130506 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.130426    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttf82\" (UniqueName: \"kubernetes.io/projected/a569873e-386e-4abd-8a51-3877414b204f-kube-api-access-ttf82\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.130506 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.130465    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a569873e-386e-4abd-8a51-3877414b204f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.130506 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.130498    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a569873e-386e-4abd-8a51-3877414b204f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.130638 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.130520    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.130638 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.130543    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a569873e-386e-4abd-8a51-3877414b204f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.130638 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.130602    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-config-volume\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.130775 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.130635    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.130775 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.130664    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.231704 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.231650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a569873e-386e-4abd-8a51-3877414b204f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.231704 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.231702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.231951 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.231734    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a569873e-386e-4abd-8a51-3877414b204f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.231951 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.231763    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-config-volume\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.231951 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.231793    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.231951 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.231818    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.231951 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.231850    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a569873e-386e-4abd-8a51-3877414b204f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.231951 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.231886    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a569873e-386e-4abd-8a51-3877414b204f-config-out\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.231951 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.231920    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.231951 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.231946    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.232347 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.231997    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-web-config\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.232347 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.232026    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttf82\" (UniqueName: \"kubernetes.io/projected/a569873e-386e-4abd-8a51-3877414b204f-kube-api-access-ttf82\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.232347 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.232054    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a569873e-386e-4abd-8a51-3877414b204f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.234385 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.234357    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a569873e-386e-4abd-8a51-3877414b204f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.234497 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.234457    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a569873e-386e-4abd-8a51-3877414b204f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.235558 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.234644    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a569873e-386e-4abd-8a51-3877414b204f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.242796 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.236323    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.242796 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.239968    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a569873e-386e-4abd-8a51-3877414b204f-config-out\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.242796 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.240426    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.242796 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.241875    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-web-config\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.242796 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.242051    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-config-volume\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.242796 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.242313    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.242796 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.242662    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a569873e-386e-4abd-8a51-3877414b204f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.243137 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.242899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.243298 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.243277    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.244746 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.244684    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttf82\" (UniqueName: \"kubernetes.io/projected/a569873e-386e-4abd-8a51-3877414b204f-kube-api-access-ttf82\") pod \"alertmanager-main-0\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.387391 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.387295    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:03:07.572910 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.572870    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:03:07.731740 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:03:07.731696    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda569873e_386e_4abd_8a51_3877414b204f.slice/crio-9cd090802af59cf541061ccd8f244b4ef896f052c8e8c5717563664f54a3c59f WatchSource:0}: Error finding container 9cd090802af59cf541061ccd8f244b4ef896f052c8e8c5717563664f54a3c59f: Status 404 returned error can't find the container with id 9cd090802af59cf541061ccd8f244b4ef896f052c8e8c5717563664f54a3c59f
Jul 27 17:03:07.927768 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.927726    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6" event={"ID":"ff30bbdb-a210-458d-b79e-be33cd423407","Type":"ContainerStarted","Data":"d93430e4f22ad769bdb717c16eff98f6d4ede2f403d2a471644f6b3b2996d4f6"}
Jul 27 17:03:07.928914 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.928888    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a569873e-386e-4abd-8a51-3877414b204f","Type":"ContainerStarted","Data":"9cd090802af59cf541061ccd8f244b4ef896f052c8e8c5717563664f54a3c59f"}
Jul 27 17:03:07.943568 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.943545    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"]
Jul 27 17:03:07.947495 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.947476    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:07.950221 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.950185    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 27 17:03:07.950320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.950216    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 27 17:03:07.950320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.950291    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 27 17:03:07.950320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.950291    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 27 17:03:07.950729 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.950692    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 27 17:03:07.950808 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.950750    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-66q8h\""
Jul 27 17:03:07.950863 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.950851    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9f409qdb04l87\""
Jul 27 17:03:07.964046 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:07.964021    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"]
Jul 27 17:03:08.041278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.041192    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04058bb3-5ad0-4102-82c5-60e7d32d445c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.041278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.041255    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04058bb3-5ad0-4102-82c5-60e7d32d445c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.041485 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.041288    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04058bb3-5ad0-4102-82c5-60e7d32d445c-metrics-client-ca\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.041485 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.041405    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04058bb3-5ad0-4102-82c5-60e7d32d445c-secret-grpc-tls\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.041485 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.041461    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dstt\" (UniqueName: \"kubernetes.io/projected/04058bb3-5ad0-4102-82c5-60e7d32d445c-kube-api-access-5dstt\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.041635 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.041543    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/04058bb3-5ad0-4102-82c5-60e7d32d445c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.041635 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.041581    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/04058bb3-5ad0-4102-82c5-60e7d32d445c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.041739 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.041655    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/04058bb3-5ad0-4102-82c5-60e7d32d445c-secret-thanos-querier-tls\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.142643 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.142590    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04058bb3-5ad0-4102-82c5-60e7d32d445c-secret-grpc-tls\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.142835 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.142667    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dstt\" (UniqueName: \"kubernetes.io/projected/04058bb3-5ad0-4102-82c5-60e7d32d445c-kube-api-access-5dstt\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.142835 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.142696    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/04058bb3-5ad0-4102-82c5-60e7d32d445c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.142835 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.142724    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/04058bb3-5ad0-4102-82c5-60e7d32d445c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.142835 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.142762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/04058bb3-5ad0-4102-82c5-60e7d32d445c-secret-thanos-querier-tls\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.142835 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.142817    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04058bb3-5ad0-4102-82c5-60e7d32d445c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.143071 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.142861    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04058bb3-5ad0-4102-82c5-60e7d32d445c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.143071 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.142890    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04058bb3-5ad0-4102-82c5-60e7d32d445c-metrics-client-ca\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.143878 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.143830    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04058bb3-5ad0-4102-82c5-60e7d32d445c-metrics-client-ca\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.145849 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.145683    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/04058bb3-5ad0-4102-82c5-60e7d32d445c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.145849 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.145760    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/04058bb3-5ad0-4102-82c5-60e7d32d445c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.146154 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.146108    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04058bb3-5ad0-4102-82c5-60e7d32d445c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.146282 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.146234    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/04058bb3-5ad0-4102-82c5-60e7d32d445c-secret-thanos-querier-tls\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.146886 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.146866    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04058bb3-5ad0-4102-82c5-60e7d32d445c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.148640 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.147552    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04058bb3-5ad0-4102-82c5-60e7d32d445c-secret-grpc-tls\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.152633 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.151633    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dstt\" (UniqueName: \"kubernetes.io/projected/04058bb3-5ad0-4102-82c5-60e7d32d445c-kube-api-access-5dstt\") pod \"thanos-querier-7dc46f647f-mr6gc\" (UID: \"04058bb3-5ad0-4102-82c5-60e7d32d445c\") " pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.258088 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.257858    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:08.439228 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.439201    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"]
Jul 27 17:03:08.477118 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:03:08.477089    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04058bb3_5ad0_4102_82c5_60e7d32d445c.slice/crio-388fc6f46c33cb980852ae5c04885870be3c5716188c1bf6b9158308ed5040b1 WatchSource:0}: Error finding container 388fc6f46c33cb980852ae5c04885870be3c5716188c1bf6b9158308ed5040b1: Status 404 returned error can't find the container with id 388fc6f46c33cb980852ae5c04885870be3c5716188c1bf6b9158308ed5040b1
Jul 27 17:03:08.934692 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.934665    2566 generic.go:356] "Generic (PLEG): container finished" podID="48f4573e-95be-44cb-b1e4-6773d04d7f68" containerID="df88e3efcd0697e29431297cd229a36f6e759cf1734c09fea5c910a06af46a6c" exitCode=0
Jul 27 17:03:08.935007 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.934747    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7m5zw" event={"ID":"48f4573e-95be-44cb-b1e4-6773d04d7f68","Type":"ContainerDied","Data":"df88e3efcd0697e29431297cd229a36f6e759cf1734c09fea5c910a06af46a6c"}
Jul 27 17:03:08.936435 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.936407    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d72jm" event={"ID":"e0651122-b4fd-4a56-8909-53661d776bfc","Type":"ContainerStarted","Data":"904ce0b27113748d8822aebfb2628ac2ea4c56a3d64d371028e4b4cdbf361ed4"}
Jul 27 17:03:08.936516 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.936464    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d72jm" event={"ID":"e0651122-b4fd-4a56-8909-53661d776bfc","Type":"ContainerStarted","Data":"d69c3edcb1b5de05ed7da5d5202b5f99f43e8cc6b28a78d9bbc4061a8467dc73"}
Jul 27 17:03:08.938471 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.938449    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6" event={"ID":"ff30bbdb-a210-458d-b79e-be33cd423407","Type":"ContainerStarted","Data":"cc332c4bf3c8a9af38b817bd8cc8ab6b5ab5438555ab573a0e6d073e4ad9f5b4"}
Jul 27 17:03:08.939434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.939416    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc" event={"ID":"04058bb3-5ad0-4102-82c5-60e7d32d445c","Type":"ContainerStarted","Data":"388fc6f46c33cb980852ae5c04885870be3c5716188c1bf6b9158308ed5040b1"}
Jul 27 17:03:08.980095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:08.980048    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-d72jm" podStartSLOduration=131.547069042 podStartE2EDuration="2m12.980032148s" totalImagesPullingTime="1.432963106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:08.979231034 +0000 UTC m=+133.054495520" watchObservedRunningTime="2026-07-27 17:03:08.980032148 +0000 UTC m=+133.055296651"
Jul 27 17:03:09.002424 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:09.002382    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dgxj6" podStartSLOduration=2.703798268 podStartE2EDuration="4.00236298s" totalImagesPullingTime="1.298564712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:09.001863385 +0000 UTC m=+133.077127873" watchObservedRunningTime="2026-07-27 17:03:09.00236298 +0000 UTC m=+133.077627469"
Jul 27 17:03:09.680533 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:09.680498    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cf44c948f-tc7r6"]
Jul 27 17:03:09.946776 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:09.946719    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7m5zw" event={"ID":"48f4573e-95be-44cb-b1e4-6773d04d7f68","Type":"ContainerStarted","Data":"739f7aad5beae874d2905b944696cd846a2a52fcf015627c4130759045d8ea90"}
Jul 27 17:03:09.947236 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:09.946796    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7m5zw" event={"ID":"48f4573e-95be-44cb-b1e4-6773d04d7f68","Type":"ContainerStarted","Data":"e6fd8ce903114b989c2d1b85932c5edd71c6c44ed9acd4192546ecdc7e38c054"}
Jul 27 17:03:09.948489 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:09.948460    2566 generic.go:356] "Generic (PLEG): container finished" podID="a569873e-386e-4abd-8a51-3877414b204f" containerID="45d40f9b282955ef412191d8146de60722ef0af3bb87394a27407313ad7bf5e5" exitCode=0
Jul 27 17:03:09.948603 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:09.948586    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a569873e-386e-4abd-8a51-3877414b204f","Type":"ContainerDied","Data":"45d40f9b282955ef412191d8146de60722ef0af3bb87394a27407313ad7bf5e5"}
Jul 27 17:03:09.964155 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:09.964062    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7m5zw" podStartSLOduration=3.481444893 podStartE2EDuration="4.964043879s" totalImagesPullingTime="1.37906868s" totalInitContainerRuntime="103.530306ms" isStatefulPod=true podCreationTimestamp="2026-07-27 17:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:09.963824251 +0000 UTC m=+134.039088738" watchObservedRunningTime="2026-07-27 17:03:09.964043879 +0000 UTC m=+134.039308351"
Jul 27 17:03:10.954137 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:10.954110    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc" event={"ID":"04058bb3-5ad0-4102-82c5-60e7d32d445c","Type":"ContainerStarted","Data":"ab94ae9e6396ac270d9a514e9825c616594fef129cc751237b4559cdf1b0dc2a"}
Jul 27 17:03:11.220113 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.220079    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"]
Jul 27 17:03:11.224890 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.224868    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.227553 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.227530    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 27 17:03:11.227672 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.227568    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 27 17:03:11.227672 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.227652    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 27 17:03:11.227672 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.227569    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-dxtq2\""
Jul 27 17:03:11.227962 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.227942    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 27 17:03:11.228032 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.228019    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 27 17:03:11.233810 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.233786    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 27 17:03:11.238303 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.238256    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"]
Jul 27 17:03:11.270470 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.270449    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwvcj\" (UniqueName: \"kubernetes.io/projected/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-kube-api-access-hwvcj\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.270592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.270477    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-secret-telemeter-client\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.270592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.270498    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-serving-certs-ca-bundle\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.270592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.270524    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-metrics-client-ca\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.270718 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.270593    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-telemeter-client-tls\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.270718 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.270622    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.270718 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.270649    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-federate-client-tls\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.270718 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.270685    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.371913 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.371870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.372095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.371927    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwvcj\" (UniqueName: \"kubernetes.io/projected/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-kube-api-access-hwvcj\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.372095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.371956    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-secret-telemeter-client\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.372095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.371980    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-serving-certs-ca-bundle\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.372095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.372018    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-metrics-client-ca\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.372095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.372061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-telemeter-client-tls\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.372403 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.372187    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.372403 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.372244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-federate-client-tls\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.372967 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.372941    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-serving-certs-ca-bundle\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.373110 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.373089    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-metrics-client-ca\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.373532 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.373510    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.374683 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.374662    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-federate-client-tls\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.374759 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.374694    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.374921 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.374905    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-secret-telemeter-client\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.374961 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.374927    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-telemeter-client-tls\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.380018 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.379998    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwvcj\" (UniqueName: \"kubernetes.io/projected/9d51eefb-51d0-42fc-83f9-249fd32b7ba5-kube-api-access-hwvcj\") pod \"telemeter-client-56ff85d9f4-q8crh\" (UID: \"9d51eefb-51d0-42fc-83f9-249fd32b7ba5\") " pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.537136 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.537060    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"
Jul 27 17:03:11.721952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.721853    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56ff85d9f4-q8crh"]
Jul 27 17:03:11.961078 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.961043    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc" event={"ID":"04058bb3-5ad0-4102-82c5-60e7d32d445c","Type":"ContainerStarted","Data":"c3d3fa173260ef93d2d5645d037c02398d0d7ab1a6c5b5a3aa6d03fa3354e9ff"}
Jul 27 17:03:11.961546 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:11.961117    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc" event={"ID":"04058bb3-5ad0-4102-82c5-60e7d32d445c","Type":"ContainerStarted","Data":"a796298dd84d47b89f55c60aa4b9f18870c12e30ac039b40060fb02bd726f43c"}
Jul 27 17:03:12.047728 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:03:12.047691    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d51eefb_51d0_42fc_83f9_249fd32b7ba5.slice/crio-303df32fdcd360b97f262e50a07b6f24153276a1942a2b4075a60d09b064d85a WatchSource:0}: Error finding container 303df32fdcd360b97f262e50a07b6f24153276a1942a2b4075a60d09b064d85a: Status 404 returned error can't find the container with id 303df32fdcd360b97f262e50a07b6f24153276a1942a2b4075a60d09b064d85a
Jul 27 17:03:12.969561 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:12.969523    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc" event={"ID":"04058bb3-5ad0-4102-82c5-60e7d32d445c","Type":"ContainerStarted","Data":"dffc996178ac1c0997e76ea4dd4ef855fea76127355e8d7a19f7c05dc495b611"}
Jul 27 17:03:12.970026 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:12.969632    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc" event={"ID":"04058bb3-5ad0-4102-82c5-60e7d32d445c","Type":"ContainerStarted","Data":"3a7abb2811d1a67fd384843b8c5d166130deb5968ce658ab875e2c2d8bd81130"}
Jul 27 17:03:12.970026 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:12.969670    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc" event={"ID":"04058bb3-5ad0-4102-82c5-60e7d32d445c","Type":"ContainerStarted","Data":"a6a28c4a6da4dcb0a5fdcf50a7ade41d76bf62cfe8288818fb0ac9f196434ae7"}
Jul 27 17:03:12.970026 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:12.969712    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:12.970857 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:12.970829    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh" event={"ID":"9d51eefb-51d0-42fc-83f9-249fd32b7ba5","Type":"ContainerStarted","Data":"303df32fdcd360b97f262e50a07b6f24153276a1942a2b4075a60d09b064d85a"}
Jul 27 17:03:12.974201 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:12.974156    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a569873e-386e-4abd-8a51-3877414b204f","Type":"ContainerStarted","Data":"7af784fca562f1034e950f30af911f2bf6e11550f25576016c3f30e80be042f3"}
Jul 27 17:03:12.974310 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:12.974233    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a569873e-386e-4abd-8a51-3877414b204f","Type":"ContainerStarted","Data":"889653b0b78ed3c1488ab46301988921a359dc16d5b2b3889ff38d644638fe40"}
Jul 27 17:03:12.974310 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:12.974262    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a569873e-386e-4abd-8a51-3877414b204f","Type":"ContainerStarted","Data":"006729003f4dd2dcc060d348e65bfb1b61374626f009959b28f5abf5f0f38632"}
Jul 27 17:03:12.974310 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:12.974287    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a569873e-386e-4abd-8a51-3877414b204f","Type":"ContainerStarted","Data":"53db21b70e3ca21af9a47cd4d81eb5962c22ecf0cf4dbe8ef2834cdebc4f7b0b"}
Jul 27 17:03:12.974420 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:12.974311    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a569873e-386e-4abd-8a51-3877414b204f","Type":"ContainerStarted","Data":"628f0053c2a9c0b0b536ea34d355cc0586bd418e5a870517b9dd6f23cbc7b821"}
Jul 27 17:03:12.974420 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:12.974339    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a569873e-386e-4abd-8a51-3877414b204f","Type":"ContainerStarted","Data":"15af2c21732c022742e3c02c3879b32e340f666e1b54c73065fda95ab60fafff"}
Jul 27 17:03:13.001952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:13.001904    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc" podStartSLOduration=2.470804932 podStartE2EDuration="6.001891487s" totalImagesPullingTime="3.531086555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:03:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:12.999520978 +0000 UTC m=+137.074785465" watchObservedRunningTime="2026-07-27 17:03:13.001891487 +0000 UTC m=+137.077155974"
Jul 27 17:03:13.031121 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:13.031080    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.53757206 podStartE2EDuration="6.031066445s" totalImagesPullingTime="3.476032731s" totalInitContainerRuntime="17.461654ms" isStatefulPod=false podCreationTimestamp="2026-07-27 17:03:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:13.028878911 +0000 UTC m=+137.104143397" watchObservedRunningTime="2026-07-27 17:03:13.031066445 +0000 UTC m=+137.106330986"
Jul 27 17:03:16.997791 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:16.997754    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh" event={"ID":"9d51eefb-51d0-42fc-83f9-249fd32b7ba5","Type":"ContainerStarted","Data":"6c773dfd9faaa70aaee0f14c469d8549101a4fbeab5ddb4d1a12efe4d283bd16"}
Jul 27 17:03:16.998228 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:16.997813    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh" event={"ID":"9d51eefb-51d0-42fc-83f9-249fd32b7ba5","Type":"ContainerStarted","Data":"4def2c451cb9510fff951a307b7feb6d07b604136138458cd3d715685989c09e"}
Jul 27 17:03:16.998228 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:16.997831    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh" event={"ID":"9d51eefb-51d0-42fc-83f9-249fd32b7ba5","Type":"ContainerStarted","Data":"cec6aa0959ff26110f2ab9e6f85ac91d526acfa8324ca50f82897f7a5fd683cd"}
Jul 27 17:03:17.029811 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:17.029756    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-56ff85d9f4-q8crh" podStartSLOduration=1.469133945 podStartE2EDuration="6.029738167s" totalImagesPullingTime="4.560604222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:17.027213258 +0000 UTC m=+141.102477744" watchObservedRunningTime="2026-07-27 17:03:17.029738167 +0000 UTC m=+141.105002655"
Jul 27 17:03:18.985758 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:18.985733    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7dc46f647f-mr6gc"
Jul 27 17:03:22.837271 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:22.837235    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bdf4764db-8r7d7"]
Jul 27 17:03:22.844789 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:22.844766    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:22.847581 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:22.847554    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 17:03:22.849072 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:22.849054    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 17:03:22.849220 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:22.849199    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bg6n7\""
Jul 27 17:03:22.849302 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:22.849062    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 17:03:22.849302 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:22.849260    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 17:03:22.849414 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:22.849145    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 17:03:22.849498 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:22.849476    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 17:03:22.849868 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:22.849848    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bdf4764db-8r7d7"]
Jul 27 17:03:22.850145 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:22.850128    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 17:03:22.973264 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:22.973231    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66ea020b-d30e-49c0-959e-328cc10fc3d7-service-ca\") pod \"console-5bdf4764db-8r7d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") " pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:22.973451 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:22.973270    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66ea020b-d30e-49c0-959e-328cc10fc3d7-oauth-serving-cert\") pod \"console-5bdf4764db-8r7d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") " pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:22.973451 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:22.973321    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66ea020b-d30e-49c0-959e-328cc10fc3d7-console-serving-cert\") pod \"console-5bdf4764db-8r7d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") " pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:22.973451 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:22.973352    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66ea020b-d30e-49c0-959e-328cc10fc3d7-console-config\") pod \"console-5bdf4764db-8r7d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") " pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:22.973605 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:22.973450    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66ea020b-d30e-49c0-959e-328cc10fc3d7-console-oauth-config\") pod \"console-5bdf4764db-8r7d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") " pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:22.973605 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:22.973498    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwhnf\" (UniqueName: \"kubernetes.io/projected/66ea020b-d30e-49c0-959e-328cc10fc3d7-kube-api-access-rwhnf\") pod \"console-5bdf4764db-8r7d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") " pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:23.073931 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:23.073898    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66ea020b-d30e-49c0-959e-328cc10fc3d7-console-oauth-config\") pod \"console-5bdf4764db-8r7d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") " pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:23.073931 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:23.073941    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwhnf\" (UniqueName: \"kubernetes.io/projected/66ea020b-d30e-49c0-959e-328cc10fc3d7-kube-api-access-rwhnf\") pod \"console-5bdf4764db-8r7d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") " pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:23.074224 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:23.073981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66ea020b-d30e-49c0-959e-328cc10fc3d7-service-ca\") pod \"console-5bdf4764db-8r7d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") " pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:23.074224 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:23.074012    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66ea020b-d30e-49c0-959e-328cc10fc3d7-oauth-serving-cert\") pod \"console-5bdf4764db-8r7d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") " pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:23.074224 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:23.074078    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66ea020b-d30e-49c0-959e-328cc10fc3d7-console-serving-cert\") pod \"console-5bdf4764db-8r7d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") " pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:23.074224 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:23.074140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66ea020b-d30e-49c0-959e-328cc10fc3d7-console-config\") pod \"console-5bdf4764db-8r7d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") " pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:23.074788 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:23.074765    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66ea020b-d30e-49c0-959e-328cc10fc3d7-service-ca\") pod \"console-5bdf4764db-8r7d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") " pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:23.074897 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:23.074877    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66ea020b-d30e-49c0-959e-328cc10fc3d7-console-config\") pod \"console-5bdf4764db-8r7d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") " pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:23.074937 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:23.074880    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66ea020b-d30e-49c0-959e-328cc10fc3d7-oauth-serving-cert\") pod \"console-5bdf4764db-8r7d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") " pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:23.076488 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:23.076466    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66ea020b-d30e-49c0-959e-328cc10fc3d7-console-serving-cert\") pod \"console-5bdf4764db-8r7d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") " pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:23.076582 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:23.076473    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66ea020b-d30e-49c0-959e-328cc10fc3d7-console-oauth-config\") pod \"console-5bdf4764db-8r7d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") " pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:23.082194 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:23.082163    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwhnf\" (UniqueName: \"kubernetes.io/projected/66ea020b-d30e-49c0-959e-328cc10fc3d7-kube-api-access-rwhnf\") pod \"console-5bdf4764db-8r7d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") " pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:23.154810 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:23.154720    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:23.282287 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:23.282264    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bdf4764db-8r7d7"]
Jul 27 17:03:23.284811 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:03:23.284784    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ea020b_d30e_49c0_959e_328cc10fc3d7.slice/crio-62c0300df3931ce49a794b81af6be0b6f57c2638cc6e8671631ce5e32e16298a WatchSource:0}: Error finding container 62c0300df3931ce49a794b81af6be0b6f57c2638cc6e8671631ce5e32e16298a: Status 404 returned error can't find the container with id 62c0300df3931ce49a794b81af6be0b6f57c2638cc6e8671631ce5e32e16298a
Jul 27 17:03:24.028057 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:24.027979    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdf4764db-8r7d7" event={"ID":"66ea020b-d30e-49c0-959e-328cc10fc3d7","Type":"ContainerStarted","Data":"62c0300df3931ce49a794b81af6be0b6f57c2638cc6e8671631ce5e32e16298a"}
Jul 27 17:03:27.044273 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:27.044241    2566 generic.go:356] "Generic (PLEG): container finished" podID="9a26877e-9895-4df2-b890-bf52e79a3239" containerID="e3fa10fd3a271f1919b468f33bffe1cd4f17bbf842f26f75fd4c2bde885108a2" exitCode=0
Jul 27 17:03:27.044644 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:27.044314    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-2vf8b" event={"ID":"9a26877e-9895-4df2-b890-bf52e79a3239","Type":"ContainerDied","Data":"e3fa10fd3a271f1919b468f33bffe1cd4f17bbf842f26f75fd4c2bde885108a2"}
Jul 27 17:03:27.044696 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:27.044670    2566 scope.go:117] "RemoveContainer" containerID="e3fa10fd3a271f1919b468f33bffe1cd4f17bbf842f26f75fd4c2bde885108a2"
Jul 27 17:03:27.045940 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:27.045917    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdf4764db-8r7d7" event={"ID":"66ea020b-d30e-49c0-959e-328cc10fc3d7","Type":"ContainerStarted","Data":"bd5506341a4493dc4c06cd4c7a1c7e3fd99aba8b3b9e731c4e9e2c0de816148b"}
Jul 27 17:03:27.083110 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:27.083064    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bdf4764db-8r7d7" podStartSLOduration=1.879177861 podStartE2EDuration="5.083047411s" totalImagesPullingTime="3.20386955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:03:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:27.082001491 +0000 UTC m=+151.157265978" watchObservedRunningTime="2026-07-27 17:03:27.083047411 +0000 UTC m=+151.158311897"
Jul 27 17:03:28.052055 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:28.052018    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-2vf8b" event={"ID":"9a26877e-9895-4df2-b890-bf52e79a3239","Type":"ContainerStarted","Data":"6995d811128ede2c624e4e76a50eb1499f93a0465bcf885a7765afa458ced29c"}
Jul 27 17:03:33.154956 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:33.154911    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:33.155356 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:33.154987    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:33.159620 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:33.159591    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:34.081598 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:34.081569    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:03:34.706463 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:34.706415    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6" podUID="79213db4-70fd-4b19-aaf6-d052590ae819" containerName="registry" containerID="cri-o://712bdd2211738a55d78a7a3fc407e6721a54f073e85e44d25ff18eebe1ff4200" gracePeriod=30
Jul 27 17:03:34.952597 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:34.952576    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:03:35.082342 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.082311    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79213db4-70fd-4b19-aaf6-d052590ae819-installation-pull-secrets\") pod \"79213db4-70fd-4b19-aaf6-d052590ae819\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") "
Jul 27 17:03:35.082527 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.082352    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79213db4-70fd-4b19-aaf6-d052590ae819-registry-certificates\") pod \"79213db4-70fd-4b19-aaf6-d052590ae819\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") "
Jul 27 17:03:35.082527 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.082373    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79213db4-70fd-4b19-aaf6-d052590ae819-trusted-ca\") pod \"79213db4-70fd-4b19-aaf6-d052590ae819\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") "
Jul 27 17:03:35.082527 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.082423    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls\") pod \"79213db4-70fd-4b19-aaf6-d052590ae819\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") "
Jul 27 17:03:35.082527 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.082443    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-bound-sa-token\") pod \"79213db4-70fd-4b19-aaf6-d052590ae819\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") "
Jul 27 17:03:35.082527 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.082469    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npm6d\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-kube-api-access-npm6d\") pod \"79213db4-70fd-4b19-aaf6-d052590ae819\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") "
Jul 27 17:03:35.082527 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.082523    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79213db4-70fd-4b19-aaf6-d052590ae819-image-registry-private-configuration\") pod \"79213db4-70fd-4b19-aaf6-d052590ae819\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") "
Jul 27 17:03:35.082835 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.082554    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79213db4-70fd-4b19-aaf6-d052590ae819-ca-trust-extracted\") pod \"79213db4-70fd-4b19-aaf6-d052590ae819\" (UID: \"79213db4-70fd-4b19-aaf6-d052590ae819\") "
Jul 27 17:03:35.082835 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.082780    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79213db4-70fd-4b19-aaf6-d052590ae819-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "79213db4-70fd-4b19-aaf6-d052590ae819" (UID: "79213db4-70fd-4b19-aaf6-d052590ae819"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:03:35.082942 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.082880    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79213db4-70fd-4b19-aaf6-d052590ae819-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "79213db4-70fd-4b19-aaf6-d052590ae819" (UID: "79213db4-70fd-4b19-aaf6-d052590ae819"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:03:35.084631 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.084107    2566 generic.go:356] "Generic (PLEG): container finished" podID="79213db4-70fd-4b19-aaf6-d052590ae819" containerID="712bdd2211738a55d78a7a3fc407e6721a54f073e85e44d25ff18eebe1ff4200" exitCode=0
Jul 27 17:03:35.084631 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.084186    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6"
Jul 27 17:03:35.084631 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.084253    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6" event={"ID":"79213db4-70fd-4b19-aaf6-d052590ae819","Type":"ContainerDied","Data":"712bdd2211738a55d78a7a3fc407e6721a54f073e85e44d25ff18eebe1ff4200"}
Jul 27 17:03:35.084631 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.084299    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cf44c948f-tc7r6" event={"ID":"79213db4-70fd-4b19-aaf6-d052590ae819","Type":"ContainerDied","Data":"69d2fc0c8e5129c9bb74ad0a3d3e24fa554e3382542c622adbfca6e402a96e33"}
Jul 27 17:03:35.084631 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.084328    2566 scope.go:117] "RemoveContainer" containerID="712bdd2211738a55d78a7a3fc407e6721a54f073e85e44d25ff18eebe1ff4200"
Jul 27 17:03:35.085314 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.085271    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "79213db4-70fd-4b19-aaf6-d052590ae819" (UID: "79213db4-70fd-4b19-aaf6-d052590ae819"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:03:35.085416 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.085368    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79213db4-70fd-4b19-aaf6-d052590ae819-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "79213db4-70fd-4b19-aaf6-d052590ae819" (UID: "79213db4-70fd-4b19-aaf6-d052590ae819"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:03:35.085533 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.085510    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-kube-api-access-npm6d" (OuterVolumeSpecName: "kube-api-access-npm6d") pod "79213db4-70fd-4b19-aaf6-d052590ae819" (UID: "79213db4-70fd-4b19-aaf6-d052590ae819"). InnerVolumeSpecName "kube-api-access-npm6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:03:35.085658 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.085636    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "79213db4-70fd-4b19-aaf6-d052590ae819" (UID: "79213db4-70fd-4b19-aaf6-d052590ae819"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:03:35.086115 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.086092    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79213db4-70fd-4b19-aaf6-d052590ae819-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "79213db4-70fd-4b19-aaf6-d052590ae819" (UID: "79213db4-70fd-4b19-aaf6-d052590ae819"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:03:35.094508 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.094483    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79213db4-70fd-4b19-aaf6-d052590ae819-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "79213db4-70fd-4b19-aaf6-d052590ae819" (UID: "79213db4-70fd-4b19-aaf6-d052590ae819"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:03:35.104149 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.104133    2566 scope.go:117] "RemoveContainer" containerID="712bdd2211738a55d78a7a3fc407e6721a54f073e85e44d25ff18eebe1ff4200"
Jul 27 17:03:35.104448 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:03:35.104427    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"712bdd2211738a55d78a7a3fc407e6721a54f073e85e44d25ff18eebe1ff4200\": container with ID starting with 712bdd2211738a55d78a7a3fc407e6721a54f073e85e44d25ff18eebe1ff4200 not found: ID does not exist" containerID="712bdd2211738a55d78a7a3fc407e6721a54f073e85e44d25ff18eebe1ff4200"
Jul 27 17:03:35.104516 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.104460    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712bdd2211738a55d78a7a3fc407e6721a54f073e85e44d25ff18eebe1ff4200"} err="failed to get container status \"712bdd2211738a55d78a7a3fc407e6721a54f073e85e44d25ff18eebe1ff4200\": rpc error: code = NotFound desc = could not find container \"712bdd2211738a55d78a7a3fc407e6721a54f073e85e44d25ff18eebe1ff4200\": container with ID starting with 712bdd2211738a55d78a7a3fc407e6721a54f073e85e44d25ff18eebe1ff4200 not found: ID does not exist"
Jul 27 17:03:35.184256 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.184221    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/79213db4-70fd-4b19-aaf6-d052590ae819-image-registry-private-configuration\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:03:35.184256 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.184260    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/79213db4-70fd-4b19-aaf6-d052590ae819-ca-trust-extracted\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:03:35.184472 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.184276    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/79213db4-70fd-4b19-aaf6-d052590ae819-installation-pull-secrets\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:03:35.184472 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.184291    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/79213db4-70fd-4b19-aaf6-d052590ae819-registry-certificates\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:03:35.184472 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.184303    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/79213db4-70fd-4b19-aaf6-d052590ae819-trusted-ca\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:03:35.184472 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.184315    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-registry-tls\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:03:35.184472 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.184325    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-bound-sa-token\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:03:35.184472 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.184337    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npm6d\" (UniqueName: \"kubernetes.io/projected/79213db4-70fd-4b19-aaf6-d052590ae819-kube-api-access-npm6d\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:03:35.406707 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.406676    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cf44c948f-tc7r6"]
Jul 27 17:03:35.413315 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:35.413289    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6cf44c948f-tc7r6"]
Jul 27 17:03:36.402737 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:36.402700    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79213db4-70fd-4b19-aaf6-d052590ae819" path="/var/lib/kubelet/pods/79213db4-70fd-4b19-aaf6-d052590ae819/volumes"
Jul 27 17:03:42.470502 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:03:42.470468    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bdf4764db-8r7d7"]
Jul 27 17:04:07.489983 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.489931    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bdf4764db-8r7d7" podUID="66ea020b-d30e-49c0-959e-328cc10fc3d7" containerName="console" containerID="cri-o://bd5506341a4493dc4c06cd4c7a1c7e3fd99aba8b3b9e731c4e9e2c0de816148b" gracePeriod=15
Jul 27 17:04:07.736319 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.736290    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:04:07.744977 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.744911    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66ea020b-d30e-49c0-959e-328cc10fc3d7-console-config\") pod \"66ea020b-d30e-49c0-959e-328cc10fc3d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") "
Jul 27 17:04:07.744977 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.744950    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66ea020b-d30e-49c0-959e-328cc10fc3d7-service-ca\") pod \"66ea020b-d30e-49c0-959e-328cc10fc3d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") "
Jul 27 17:04:07.745149 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.744975    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66ea020b-d30e-49c0-959e-328cc10fc3d7-oauth-serving-cert\") pod \"66ea020b-d30e-49c0-959e-328cc10fc3d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") "
Jul 27 17:04:07.745149 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.745006    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66ea020b-d30e-49c0-959e-328cc10fc3d7-console-serving-cert\") pod \"66ea020b-d30e-49c0-959e-328cc10fc3d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") "
Jul 27 17:04:07.745149 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.745049    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66ea020b-d30e-49c0-959e-328cc10fc3d7-console-oauth-config\") pod \"66ea020b-d30e-49c0-959e-328cc10fc3d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") "
Jul 27 17:04:07.745149 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.745119    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwhnf\" (UniqueName: \"kubernetes.io/projected/66ea020b-d30e-49c0-959e-328cc10fc3d7-kube-api-access-rwhnf\") pod \"66ea020b-d30e-49c0-959e-328cc10fc3d7\" (UID: \"66ea020b-d30e-49c0-959e-328cc10fc3d7\") "
Jul 27 17:04:07.745382 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.745356    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ea020b-d30e-49c0-959e-328cc10fc3d7-service-ca" (OuterVolumeSpecName: "service-ca") pod "66ea020b-d30e-49c0-959e-328cc10fc3d7" (UID: "66ea020b-d30e-49c0-959e-328cc10fc3d7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:04:07.745443 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.745363    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ea020b-d30e-49c0-959e-328cc10fc3d7-console-config" (OuterVolumeSpecName: "console-config") pod "66ea020b-d30e-49c0-959e-328cc10fc3d7" (UID: "66ea020b-d30e-49c0-959e-328cc10fc3d7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:04:07.745443 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.745433    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ea020b-d30e-49c0-959e-328cc10fc3d7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "66ea020b-d30e-49c0-959e-328cc10fc3d7" (UID: "66ea020b-d30e-49c0-959e-328cc10fc3d7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:04:07.747359 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.747332    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ea020b-d30e-49c0-959e-328cc10fc3d7-kube-api-access-rwhnf" (OuterVolumeSpecName: "kube-api-access-rwhnf") pod "66ea020b-d30e-49c0-959e-328cc10fc3d7" (UID: "66ea020b-d30e-49c0-959e-328cc10fc3d7"). InnerVolumeSpecName "kube-api-access-rwhnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:04:07.747438 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.747400    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ea020b-d30e-49c0-959e-328cc10fc3d7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "66ea020b-d30e-49c0-959e-328cc10fc3d7" (UID: "66ea020b-d30e-49c0-959e-328cc10fc3d7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:04:07.747474 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.747419    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ea020b-d30e-49c0-959e-328cc10fc3d7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "66ea020b-d30e-49c0-959e-328cc10fc3d7" (UID: "66ea020b-d30e-49c0-959e-328cc10fc3d7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:04:07.846545 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.846506    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66ea020b-d30e-49c0-959e-328cc10fc3d7-console-config\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:04:07.846545 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.846536    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66ea020b-d30e-49c0-959e-328cc10fc3d7-service-ca\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:04:07.846545 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.846545    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66ea020b-d30e-49c0-959e-328cc10fc3d7-oauth-serving-cert\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:04:07.846545 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.846555    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66ea020b-d30e-49c0-959e-328cc10fc3d7-console-serving-cert\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:04:07.846793 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.846563    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66ea020b-d30e-49c0-959e-328cc10fc3d7-console-oauth-config\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:04:07.846793 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:07.846572    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwhnf\" (UniqueName: \"kubernetes.io/projected/66ea020b-d30e-49c0-959e-328cc10fc3d7-kube-api-access-rwhnf\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:04:08.225461 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:08.225423    2566 generic.go:356] "Generic (PLEG): container finished" podID="66ea020b-d30e-49c0-959e-328cc10fc3d7" containerID="bd5506341a4493dc4c06cd4c7a1c7e3fd99aba8b3b9e731c4e9e2c0de816148b" exitCode=0
Jul 27 17:04:08.225634 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:08.225468    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdf4764db-8r7d7"
Jul 27 17:04:08.225634 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:08.225473    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdf4764db-8r7d7" event={"ID":"66ea020b-d30e-49c0-959e-328cc10fc3d7","Type":"ContainerDied","Data":"bd5506341a4493dc4c06cd4c7a1c7e3fd99aba8b3b9e731c4e9e2c0de816148b"}
Jul 27 17:04:08.225634 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:08.225548    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdf4764db-8r7d7" event={"ID":"66ea020b-d30e-49c0-959e-328cc10fc3d7","Type":"ContainerDied","Data":"62c0300df3931ce49a794b81af6be0b6f57c2638cc6e8671631ce5e32e16298a"}
Jul 27 17:04:08.225634 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:08.225569    2566 scope.go:117] "RemoveContainer" containerID="bd5506341a4493dc4c06cd4c7a1c7e3fd99aba8b3b9e731c4e9e2c0de816148b"
Jul 27 17:04:08.234138 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:08.234119    2566 scope.go:117] "RemoveContainer" containerID="bd5506341a4493dc4c06cd4c7a1c7e3fd99aba8b3b9e731c4e9e2c0de816148b"
Jul 27 17:04:08.234401 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:04:08.234383    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd5506341a4493dc4c06cd4c7a1c7e3fd99aba8b3b9e731c4e9e2c0de816148b\": container with ID starting with bd5506341a4493dc4c06cd4c7a1c7e3fd99aba8b3b9e731c4e9e2c0de816148b not found: ID does not exist" containerID="bd5506341a4493dc4c06cd4c7a1c7e3fd99aba8b3b9e731c4e9e2c0de816148b"
Jul 27 17:04:08.234457 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:08.234409    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5506341a4493dc4c06cd4c7a1c7e3fd99aba8b3b9e731c4e9e2c0de816148b"} err="failed to get container status \"bd5506341a4493dc4c06cd4c7a1c7e3fd99aba8b3b9e731c4e9e2c0de816148b\": rpc error: code = NotFound desc = could not find container \"bd5506341a4493dc4c06cd4c7a1c7e3fd99aba8b3b9e731c4e9e2c0de816148b\": container with ID starting with bd5506341a4493dc4c06cd4c7a1c7e3fd99aba8b3b9e731c4e9e2c0de816148b not found: ID does not exist"
Jul 27 17:04:08.246463 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:08.246441    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bdf4764db-8r7d7"]
Jul 27 17:04:08.247741 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:08.247723    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bdf4764db-8r7d7"]
Jul 27 17:04:08.402105 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:08.402071    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66ea020b-d30e-49c0-959e-328cc10fc3d7" path="/var/lib/kubelet/pods/66ea020b-d30e-49c0-959e-328cc10fc3d7/volumes"
Jul 27 17:04:26.463676 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:26.463634    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:04:26.464198 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:26.464040    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="alertmanager" containerID="cri-o://15af2c21732c022742e3c02c3879b32e340f666e1b54c73065fda95ab60fafff" gracePeriod=120
Jul 27 17:04:26.464198 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:26.464060    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="kube-rbac-proxy-metric" containerID="cri-o://889653b0b78ed3c1488ab46301988921a359dc16d5b2b3889ff38d644638fe40" gracePeriod=120
Jul 27 17:04:26.464198 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:26.464091    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="config-reloader" containerID="cri-o://628f0053c2a9c0b0b536ea34d355cc0586bd418e5a870517b9dd6f23cbc7b821" gracePeriod=120
Jul 27 17:04:26.464198 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:26.464119    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="kube-rbac-proxy" containerID="cri-o://006729003f4dd2dcc060d348e65bfb1b61374626f009959b28f5abf5f0f38632" gracePeriod=120
Jul 27 17:04:26.464198 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:26.464074    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="kube-rbac-proxy-web" containerID="cri-o://53db21b70e3ca21af9a47cd4d81eb5962c22ecf0cf4dbe8ef2834cdebc4f7b0b" gracePeriod=120
Jul 27 17:04:26.464471 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:26.464101    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="prom-label-proxy" containerID="cri-o://7af784fca562f1034e950f30af911f2bf6e11550f25576016c3f30e80be042f3" gracePeriod=120
Jul 27 17:04:27.306451 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.306419    2566 generic.go:356] "Generic (PLEG): container finished" podID="a569873e-386e-4abd-8a51-3877414b204f" containerID="7af784fca562f1034e950f30af911f2bf6e11550f25576016c3f30e80be042f3" exitCode=0
Jul 27 17:04:27.306451 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.306447    2566 generic.go:356] "Generic (PLEG): container finished" podID="a569873e-386e-4abd-8a51-3877414b204f" containerID="889653b0b78ed3c1488ab46301988921a359dc16d5b2b3889ff38d644638fe40" exitCode=0
Jul 27 17:04:27.306451 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.306455    2566 generic.go:356] "Generic (PLEG): container finished" podID="a569873e-386e-4abd-8a51-3877414b204f" containerID="006729003f4dd2dcc060d348e65bfb1b61374626f009959b28f5abf5f0f38632" exitCode=0
Jul 27 17:04:27.306451 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.306461    2566 generic.go:356] "Generic (PLEG): container finished" podID="a569873e-386e-4abd-8a51-3877414b204f" containerID="628f0053c2a9c0b0b536ea34d355cc0586bd418e5a870517b9dd6f23cbc7b821" exitCode=0
Jul 27 17:04:27.306713 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.306467    2566 generic.go:356] "Generic (PLEG): container finished" podID="a569873e-386e-4abd-8a51-3877414b204f" containerID="15af2c21732c022742e3c02c3879b32e340f666e1b54c73065fda95ab60fafff" exitCode=0
Jul 27 17:04:27.306713 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.306491    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a569873e-386e-4abd-8a51-3877414b204f","Type":"ContainerDied","Data":"7af784fca562f1034e950f30af911f2bf6e11550f25576016c3f30e80be042f3"}
Jul 27 17:04:27.306713 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.306554    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a569873e-386e-4abd-8a51-3877414b204f","Type":"ContainerDied","Data":"889653b0b78ed3c1488ab46301988921a359dc16d5b2b3889ff38d644638fe40"}
Jul 27 17:04:27.306713 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.306581    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a569873e-386e-4abd-8a51-3877414b204f","Type":"ContainerDied","Data":"006729003f4dd2dcc060d348e65bfb1b61374626f009959b28f5abf5f0f38632"}
Jul 27 17:04:27.306713 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.306606    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a569873e-386e-4abd-8a51-3877414b204f","Type":"ContainerDied","Data":"628f0053c2a9c0b0b536ea34d355cc0586bd418e5a870517b9dd6f23cbc7b821"}
Jul 27 17:04:27.306713 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.306628    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a569873e-386e-4abd-8a51-3877414b204f","Type":"ContainerDied","Data":"15af2c21732c022742e3c02c3879b32e340f666e1b54c73065fda95ab60fafff"}
Jul 27 17:04:27.716373 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.716341    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:27.812866 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.812839    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-web-config\") pod \"a569873e-386e-4abd-8a51-3877414b204f\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") "
Jul 27 17:04:27.813051 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.812875    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a569873e-386e-4abd-8a51-3877414b204f-config-out\") pod \"a569873e-386e-4abd-8a51-3877414b204f\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") "
Jul 27 17:04:27.813051 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.812896    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttf82\" (UniqueName: \"kubernetes.io/projected/a569873e-386e-4abd-8a51-3877414b204f-kube-api-access-ttf82\") pod \"a569873e-386e-4abd-8a51-3877414b204f\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") "
Jul 27 17:04:27.813051 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.812917    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a569873e-386e-4abd-8a51-3877414b204f-alertmanager-main-db\") pod \"a569873e-386e-4abd-8a51-3877414b204f\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") "
Jul 27 17:04:27.813051 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.812936    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a569873e-386e-4abd-8a51-3877414b204f-metrics-client-ca\") pod \"a569873e-386e-4abd-8a51-3877414b204f\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") "
Jul 27 17:04:27.813051 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.812980    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a569873e-386e-4abd-8a51-3877414b204f-tls-assets\") pod \"a569873e-386e-4abd-8a51-3877414b204f\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") "
Jul 27 17:04:27.813051 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.813007    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a569873e-386e-4abd-8a51-3877414b204f\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") "
Jul 27 17:04:27.813051 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.813036    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-kube-rbac-proxy\") pod \"a569873e-386e-4abd-8a51-3877414b204f\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") "
Jul 27 17:04:27.813429 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.813098    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-cluster-tls-config\") pod \"a569873e-386e-4abd-8a51-3877414b204f\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") "
Jul 27 17:04:27.813429 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.813124    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-main-tls\") pod \"a569873e-386e-4abd-8a51-3877414b204f\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") "
Jul 27 17:04:27.813429 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.813151    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a569873e-386e-4abd-8a51-3877414b204f-alertmanager-trusted-ca-bundle\") pod \"a569873e-386e-4abd-8a51-3877414b204f\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") "
Jul 27 17:04:27.813429 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.813216    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-kube-rbac-proxy-web\") pod \"a569873e-386e-4abd-8a51-3877414b204f\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") "
Jul 27 17:04:27.813429 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.813323    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-config-volume\") pod \"a569873e-386e-4abd-8a51-3877414b204f\" (UID: \"a569873e-386e-4abd-8a51-3877414b204f\") "
Jul 27 17:04:27.813676 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.813480    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a569873e-386e-4abd-8a51-3877414b204f-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "a569873e-386e-4abd-8a51-3877414b204f" (UID: "a569873e-386e-4abd-8a51-3877414b204f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:04:27.813676 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.813573    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a569873e-386e-4abd-8a51-3877414b204f-metrics-client-ca\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:04:27.814152 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.814009    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a569873e-386e-4abd-8a51-3877414b204f-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "a569873e-386e-4abd-8a51-3877414b204f" (UID: "a569873e-386e-4abd-8a51-3877414b204f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:04:27.814238 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.814206    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a569873e-386e-4abd-8a51-3877414b204f-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "a569873e-386e-4abd-8a51-3877414b204f" (UID: "a569873e-386e-4abd-8a51-3877414b204f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:04:27.815927 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.815885    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a569873e-386e-4abd-8a51-3877414b204f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a569873e-386e-4abd-8a51-3877414b204f" (UID: "a569873e-386e-4abd-8a51-3877414b204f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:04:27.815927 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.815904    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a569873e-386e-4abd-8a51-3877414b204f-kube-api-access-ttf82" (OuterVolumeSpecName: "kube-api-access-ttf82") pod "a569873e-386e-4abd-8a51-3877414b204f" (UID: "a569873e-386e-4abd-8a51-3877414b204f"). InnerVolumeSpecName "kube-api-access-ttf82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:04:27.816270 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.816224    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "a569873e-386e-4abd-8a51-3877414b204f" (UID: "a569873e-386e-4abd-8a51-3877414b204f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:04:27.816602 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.816575    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "a569873e-386e-4abd-8a51-3877414b204f" (UID: "a569873e-386e-4abd-8a51-3877414b204f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:04:27.816716 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.816641    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "a569873e-386e-4abd-8a51-3877414b204f" (UID: "a569873e-386e-4abd-8a51-3877414b204f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:04:27.816846 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.816822    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a569873e-386e-4abd-8a51-3877414b204f-config-out" (OuterVolumeSpecName: "config-out") pod "a569873e-386e-4abd-8a51-3877414b204f" (UID: "a569873e-386e-4abd-8a51-3877414b204f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:04:27.817089 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.817067    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "a569873e-386e-4abd-8a51-3877414b204f" (UID: "a569873e-386e-4abd-8a51-3877414b204f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:04:27.817647 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.817628    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-config-volume" (OuterVolumeSpecName: "config-volume") pod "a569873e-386e-4abd-8a51-3877414b204f" (UID: "a569873e-386e-4abd-8a51-3877414b204f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:04:27.820447 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.820390    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "a569873e-386e-4abd-8a51-3877414b204f" (UID: "a569873e-386e-4abd-8a51-3877414b204f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:04:27.827472 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.827446    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-web-config" (OuterVolumeSpecName: "web-config") pod "a569873e-386e-4abd-8a51-3877414b204f" (UID: "a569873e-386e-4abd-8a51-3877414b204f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:04:27.914741 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.914692    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:04:27.914741 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.914737    2566 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-config-volume\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:04:27.914741 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.914749    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-web-config\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:04:27.914741 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.914756    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a569873e-386e-4abd-8a51-3877414b204f-config-out\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:04:27.915008 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.914766    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttf82\" (UniqueName: \"kubernetes.io/projected/a569873e-386e-4abd-8a51-3877414b204f-kube-api-access-ttf82\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:04:27.915008 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.914778    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a569873e-386e-4abd-8a51-3877414b204f-alertmanager-main-db\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:04:27.915008 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.914786    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a569873e-386e-4abd-8a51-3877414b204f-tls-assets\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:04:27.915008 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.914795    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:04:27.915008 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.914804    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:04:27.915008 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.914813    2566 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-cluster-tls-config\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:04:27.915008 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.914821    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a569873e-386e-4abd-8a51-3877414b204f-secret-alertmanager-main-tls\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:04:27.915008 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:27.914829    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a569873e-386e-4abd-8a51-3877414b204f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:04:28.313347 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.313307    2566 generic.go:356] "Generic (PLEG): container finished" podID="a569873e-386e-4abd-8a51-3877414b204f" containerID="53db21b70e3ca21af9a47cd4d81eb5962c22ecf0cf4dbe8ef2834cdebc4f7b0b" exitCode=0
Jul 27 17:04:28.313347 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.313350    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a569873e-386e-4abd-8a51-3877414b204f","Type":"ContainerDied","Data":"53db21b70e3ca21af9a47cd4d81eb5962c22ecf0cf4dbe8ef2834cdebc4f7b0b"}
Jul 27 17:04:28.313559 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.313392    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.313559 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.313411    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a569873e-386e-4abd-8a51-3877414b204f","Type":"ContainerDied","Data":"9cd090802af59cf541061ccd8f244b4ef896f052c8e8c5717563664f54a3c59f"}
Jul 27 17:04:28.313559 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.313446    2566 scope.go:117] "RemoveContainer" containerID="7af784fca562f1034e950f30af911f2bf6e11550f25576016c3f30e80be042f3"
Jul 27 17:04:28.321156 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.321136    2566 scope.go:117] "RemoveContainer" containerID="889653b0b78ed3c1488ab46301988921a359dc16d5b2b3889ff38d644638fe40"
Jul 27 17:04:28.333662 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.333643    2566 scope.go:117] "RemoveContainer" containerID="006729003f4dd2dcc060d348e65bfb1b61374626f009959b28f5abf5f0f38632"
Jul 27 17:04:28.340476 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.340458    2566 scope.go:117] "RemoveContainer" containerID="53db21b70e3ca21af9a47cd4d81eb5962c22ecf0cf4dbe8ef2834cdebc4f7b0b"
Jul 27 17:04:28.343242 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.343204    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:04:28.347853 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.347829    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:04:28.348540 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.348506    2566 scope.go:117] "RemoveContainer" containerID="628f0053c2a9c0b0b536ea34d355cc0586bd418e5a870517b9dd6f23cbc7b821"
Jul 27 17:04:28.355550 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.355532    2566 scope.go:117] "RemoveContainer" containerID="15af2c21732c022742e3c02c3879b32e340f666e1b54c73065fda95ab60fafff"
Jul 27 17:04:28.362793 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.362773    2566 scope.go:117] "RemoveContainer" containerID="45d40f9b282955ef412191d8146de60722ef0af3bb87394a27407313ad7bf5e5"
Jul 27 17:04:28.369882 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.369862    2566 scope.go:117] "RemoveContainer" containerID="7af784fca562f1034e950f30af911f2bf6e11550f25576016c3f30e80be042f3"
Jul 27 17:04:28.370132 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:04:28.370115    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af784fca562f1034e950f30af911f2bf6e11550f25576016c3f30e80be042f3\": container with ID starting with 7af784fca562f1034e950f30af911f2bf6e11550f25576016c3f30e80be042f3 not found: ID does not exist" containerID="7af784fca562f1034e950f30af911f2bf6e11550f25576016c3f30e80be042f3"
Jul 27 17:04:28.370206 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.370140    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af784fca562f1034e950f30af911f2bf6e11550f25576016c3f30e80be042f3"} err="failed to get container status \"7af784fca562f1034e950f30af911f2bf6e11550f25576016c3f30e80be042f3\": rpc error: code = NotFound desc = could not find container \"7af784fca562f1034e950f30af911f2bf6e11550f25576016c3f30e80be042f3\": container with ID starting with 7af784fca562f1034e950f30af911f2bf6e11550f25576016c3f30e80be042f3 not found: ID does not exist"
Jul 27 17:04:28.370206 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.370158    2566 scope.go:117] "RemoveContainer" containerID="889653b0b78ed3c1488ab46301988921a359dc16d5b2b3889ff38d644638fe40"
Jul 27 17:04:28.370416 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:04:28.370388    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"889653b0b78ed3c1488ab46301988921a359dc16d5b2b3889ff38d644638fe40\": container with ID starting with 889653b0b78ed3c1488ab46301988921a359dc16d5b2b3889ff38d644638fe40 not found: ID does not exist" containerID="889653b0b78ed3c1488ab46301988921a359dc16d5b2b3889ff38d644638fe40"
Jul 27 17:04:28.370456 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.370423    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"889653b0b78ed3c1488ab46301988921a359dc16d5b2b3889ff38d644638fe40"} err="failed to get container status \"889653b0b78ed3c1488ab46301988921a359dc16d5b2b3889ff38d644638fe40\": rpc error: code = NotFound desc = could not find container \"889653b0b78ed3c1488ab46301988921a359dc16d5b2b3889ff38d644638fe40\": container with ID starting with 889653b0b78ed3c1488ab46301988921a359dc16d5b2b3889ff38d644638fe40 not found: ID does not exist"
Jul 27 17:04:28.370456 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.370440    2566 scope.go:117] "RemoveContainer" containerID="006729003f4dd2dcc060d348e65bfb1b61374626f009959b28f5abf5f0f38632"
Jul 27 17:04:28.370657 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:04:28.370640    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"006729003f4dd2dcc060d348e65bfb1b61374626f009959b28f5abf5f0f38632\": container with ID starting with 006729003f4dd2dcc060d348e65bfb1b61374626f009959b28f5abf5f0f38632 not found: ID does not exist" containerID="006729003f4dd2dcc060d348e65bfb1b61374626f009959b28f5abf5f0f38632"
Jul 27 17:04:28.370695 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.370661    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"006729003f4dd2dcc060d348e65bfb1b61374626f009959b28f5abf5f0f38632"} err="failed to get container status \"006729003f4dd2dcc060d348e65bfb1b61374626f009959b28f5abf5f0f38632\": rpc error: code = NotFound desc = could not find container \"006729003f4dd2dcc060d348e65bfb1b61374626f009959b28f5abf5f0f38632\": container with ID starting with 006729003f4dd2dcc060d348e65bfb1b61374626f009959b28f5abf5f0f38632 not found: ID does not exist"
Jul 27 17:04:28.370695 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.370674    2566 scope.go:117] "RemoveContainer" containerID="53db21b70e3ca21af9a47cd4d81eb5962c22ecf0cf4dbe8ef2834cdebc4f7b0b"
Jul 27 17:04:28.370871 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:04:28.370856    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53db21b70e3ca21af9a47cd4d81eb5962c22ecf0cf4dbe8ef2834cdebc4f7b0b\": container with ID starting with 53db21b70e3ca21af9a47cd4d81eb5962c22ecf0cf4dbe8ef2834cdebc4f7b0b not found: ID does not exist" containerID="53db21b70e3ca21af9a47cd4d81eb5962c22ecf0cf4dbe8ef2834cdebc4f7b0b"
Jul 27 17:04:28.370917 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.370873    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53db21b70e3ca21af9a47cd4d81eb5962c22ecf0cf4dbe8ef2834cdebc4f7b0b"} err="failed to get container status \"53db21b70e3ca21af9a47cd4d81eb5962c22ecf0cf4dbe8ef2834cdebc4f7b0b\": rpc error: code = NotFound desc = could not find container \"53db21b70e3ca21af9a47cd4d81eb5962c22ecf0cf4dbe8ef2834cdebc4f7b0b\": container with ID starting with 53db21b70e3ca21af9a47cd4d81eb5962c22ecf0cf4dbe8ef2834cdebc4f7b0b not found: ID does not exist"
Jul 27 17:04:28.370917 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.370886    2566 scope.go:117] "RemoveContainer" containerID="628f0053c2a9c0b0b536ea34d355cc0586bd418e5a870517b9dd6f23cbc7b821"
Jul 27 17:04:28.371124 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:04:28.371110    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"628f0053c2a9c0b0b536ea34d355cc0586bd418e5a870517b9dd6f23cbc7b821\": container with ID starting with 628f0053c2a9c0b0b536ea34d355cc0586bd418e5a870517b9dd6f23cbc7b821 not found: ID does not exist" containerID="628f0053c2a9c0b0b536ea34d355cc0586bd418e5a870517b9dd6f23cbc7b821"
Jul 27 17:04:28.371159 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.371130    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"628f0053c2a9c0b0b536ea34d355cc0586bd418e5a870517b9dd6f23cbc7b821"} err="failed to get container status \"628f0053c2a9c0b0b536ea34d355cc0586bd418e5a870517b9dd6f23cbc7b821\": rpc error: code = NotFound desc = could not find container \"628f0053c2a9c0b0b536ea34d355cc0586bd418e5a870517b9dd6f23cbc7b821\": container with ID starting with 628f0053c2a9c0b0b536ea34d355cc0586bd418e5a870517b9dd6f23cbc7b821 not found: ID does not exist"
Jul 27 17:04:28.371159 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.371142    2566 scope.go:117] "RemoveContainer" containerID="15af2c21732c022742e3c02c3879b32e340f666e1b54c73065fda95ab60fafff"
Jul 27 17:04:28.371389 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:04:28.371370    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15af2c21732c022742e3c02c3879b32e340f666e1b54c73065fda95ab60fafff\": container with ID starting with 15af2c21732c022742e3c02c3879b32e340f666e1b54c73065fda95ab60fafff not found: ID does not exist" containerID="15af2c21732c022742e3c02c3879b32e340f666e1b54c73065fda95ab60fafff"
Jul 27 17:04:28.371434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.371394    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15af2c21732c022742e3c02c3879b32e340f666e1b54c73065fda95ab60fafff"} err="failed to get container status \"15af2c21732c022742e3c02c3879b32e340f666e1b54c73065fda95ab60fafff\": rpc error: code = NotFound desc = could not find container \"15af2c21732c022742e3c02c3879b32e340f666e1b54c73065fda95ab60fafff\": container with ID starting with 15af2c21732c022742e3c02c3879b32e340f666e1b54c73065fda95ab60fafff not found: ID does not exist"
Jul 27 17:04:28.371434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.371408    2566 scope.go:117] "RemoveContainer" containerID="45d40f9b282955ef412191d8146de60722ef0af3bb87394a27407313ad7bf5e5"
Jul 27 17:04:28.371660 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:04:28.371643    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d40f9b282955ef412191d8146de60722ef0af3bb87394a27407313ad7bf5e5\": container with ID starting with 45d40f9b282955ef412191d8146de60722ef0af3bb87394a27407313ad7bf5e5 not found: ID does not exist" containerID="45d40f9b282955ef412191d8146de60722ef0af3bb87394a27407313ad7bf5e5"
Jul 27 17:04:28.371707 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.371668    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d40f9b282955ef412191d8146de60722ef0af3bb87394a27407313ad7bf5e5"} err="failed to get container status \"45d40f9b282955ef412191d8146de60722ef0af3bb87394a27407313ad7bf5e5\": rpc error: code = NotFound desc = could not find container \"45d40f9b282955ef412191d8146de60722ef0af3bb87394a27407313ad7bf5e5\": container with ID starting with 45d40f9b282955ef412191d8146de60722ef0af3bb87394a27407313ad7bf5e5 not found: ID does not exist"
Jul 27 17:04:28.373581 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.373561    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:04:28.373890 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.373876    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="kube-rbac-proxy"
Jul 27 17:04:28.373931 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.373892    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="kube-rbac-proxy"
Jul 27 17:04:28.373931 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.373905    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79213db4-70fd-4b19-aaf6-d052590ae819" containerName="registry"
Jul 27 17:04:28.373931 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.373911    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="79213db4-70fd-4b19-aaf6-d052590ae819" containerName="registry"
Jul 27 17:04:28.373931 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.373920    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="prom-label-proxy"
Jul 27 17:04:28.373931 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.373925    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="prom-label-proxy"
Jul 27 17:04:28.373931 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.373932    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="init-config-reloader"
Jul 27 17:04:28.374095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.373937    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="init-config-reloader"
Jul 27 17:04:28.374095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.373945    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="alertmanager"
Jul 27 17:04:28.374095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.373950    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="alertmanager"
Jul 27 17:04:28.374095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.373956    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="kube-rbac-proxy-web"
Jul 27 17:04:28.374095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.373961    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="kube-rbac-proxy-web"
Jul 27 17:04:28.374095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.373973    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="kube-rbac-proxy-metric"
Jul 27 17:04:28.374095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.373979    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="kube-rbac-proxy-metric"
Jul 27 17:04:28.374095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.373984    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ea020b-d30e-49c0-959e-328cc10fc3d7" containerName="console"
Jul 27 17:04:28.374095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.373989    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ea020b-d30e-49c0-959e-328cc10fc3d7" containerName="console"
Jul 27 17:04:28.374095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.373999    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="config-reloader"
Jul 27 17:04:28.374095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.374004    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="config-reloader"
Jul 27 17:04:28.374095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.374045    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="kube-rbac-proxy-metric"
Jul 27 17:04:28.374095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.374052    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="alertmanager"
Jul 27 17:04:28.374095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.374058    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="config-reloader"
Jul 27 17:04:28.374095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.374063    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="79213db4-70fd-4b19-aaf6-d052590ae819" containerName="registry"
Jul 27 17:04:28.374095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.374068    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="prom-label-proxy"
Jul 27 17:04:28.374095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.374074    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="kube-rbac-proxy"
Jul 27 17:04:28.374095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.374079    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a569873e-386e-4abd-8a51-3877414b204f" containerName="kube-rbac-proxy-web"
Jul 27 17:04:28.374095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.374086    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ea020b-d30e-49c0-959e-328cc10fc3d7" containerName="console"
Jul 27 17:04:28.379927 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.379908    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.382504 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.382486    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 17:04:28.382650 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.382480    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 17:04:28.382742 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.382492    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 17:04:28.382742 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.382691    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 17:04:28.382742 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.382495    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 17:04:28.382742 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.382564    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m77t2\""
Jul 27 17:04:28.383019 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.382914    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 17:04:28.383091 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.383069    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 17:04:28.383200 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.383162    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 17:04:28.388411 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.388393    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 17:04:28.397562 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.397530    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:04:28.403612 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.403584    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a569873e-386e-4abd-8a51-3877414b204f" path="/var/lib/kubelet/pods/a569873e-386e-4abd-8a51-3877414b204f/volumes"
Jul 27 17:04:28.519747 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.519700    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-web-config\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.519747 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.519749    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e906720c-c8d1-4be6-9f5f-82bf93864fb0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.519955 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.519774    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e906720c-c8d1-4be6-9f5f-82bf93864fb0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.519955 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.519852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.519955 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.519882    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.519955 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.519904    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e906720c-c8d1-4be6-9f5f-82bf93864fb0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.519955 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.519921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e906720c-c8d1-4be6-9f5f-82bf93864fb0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.520106 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.519967    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.520106 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.520006    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-config-volume\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.520106 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.520035    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.520106 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.520062    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e906720c-c8d1-4be6-9f5f-82bf93864fb0-config-out\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.520106 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.520079    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.520106 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.520104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66tlp\" (UniqueName: \"kubernetes.io/projected/e906720c-c8d1-4be6-9f5f-82bf93864fb0-kube-api-access-66tlp\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.621023 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.620984    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.621023 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.621031    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e906720c-c8d1-4be6-9f5f-82bf93864fb0-config-out\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.621268 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.621049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.621268 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.621066    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66tlp\" (UniqueName: \"kubernetes.io/projected/e906720c-c8d1-4be6-9f5f-82bf93864fb0-kube-api-access-66tlp\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.621367 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.621284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-web-config\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.621367 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.621326    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e906720c-c8d1-4be6-9f5f-82bf93864fb0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.621367 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.621357    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e906720c-c8d1-4be6-9f5f-82bf93864fb0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.621523 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.621396    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.621523 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.621422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.621523 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.621452    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e906720c-c8d1-4be6-9f5f-82bf93864fb0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.621523 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.621474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e906720c-c8d1-4be6-9f5f-82bf93864fb0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.621722 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.621520    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.621722 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.621584    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-config-volume\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.621722 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.621707    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e906720c-c8d1-4be6-9f5f-82bf93864fb0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.622473 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.622440    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e906720c-c8d1-4be6-9f5f-82bf93864fb0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.624082 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.624053    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e906720c-c8d1-4be6-9f5f-82bf93864fb0-config-out\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.624581 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.624333    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.624581 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.624384    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.624581 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.624534    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e906720c-c8d1-4be6-9f5f-82bf93864fb0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.624581 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.624560    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-config-volume\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.624857 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.624809    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-web-config\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.625206 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.625158    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e906720c-c8d1-4be6-9f5f-82bf93864fb0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.625606 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.625584    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.625649 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.625635    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.626209 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.626193    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e906720c-c8d1-4be6-9f5f-82bf93864fb0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.628989 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.628968    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66tlp\" (UniqueName: \"kubernetes.io/projected/e906720c-c8d1-4be6-9f5f-82bf93864fb0-kube-api-access-66tlp\") pod \"alertmanager-main-0\" (UID: \"e906720c-c8d1-4be6-9f5f-82bf93864fb0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.690826 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.690793    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:04:28.848179 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:28.848142    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:04:28.850161 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:04:28.850126    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode906720c_c8d1_4be6_9f5f_82bf93864fb0.slice/crio-916790c500a421c1cd157c8c098079600c62d7bb34a4b250f54fd77261170f2a WatchSource:0}: Error finding container 916790c500a421c1cd157c8c098079600c62d7bb34a4b250f54fd77261170f2a: Status 404 returned error can't find the container with id 916790c500a421c1cd157c8c098079600c62d7bb34a4b250f54fd77261170f2a
Jul 27 17:04:29.319486 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:29.319449    2566 generic.go:356] "Generic (PLEG): container finished" podID="e906720c-c8d1-4be6-9f5f-82bf93864fb0" containerID="d369e9fbfc24e5aca48fa1f04e36807b04707b1ec2c7607b39a416dfb3a5d9e4" exitCode=0
Jul 27 17:04:29.319651 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:29.319538    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e906720c-c8d1-4be6-9f5f-82bf93864fb0","Type":"ContainerDied","Data":"d369e9fbfc24e5aca48fa1f04e36807b04707b1ec2c7607b39a416dfb3a5d9e4"}
Jul 27 17:04:29.319651 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:29.319608    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e906720c-c8d1-4be6-9f5f-82bf93864fb0","Type":"ContainerStarted","Data":"916790c500a421c1cd157c8c098079600c62d7bb34a4b250f54fd77261170f2a"}
Jul 27 17:04:30.332123 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:30.332068    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e906720c-c8d1-4be6-9f5f-82bf93864fb0","Type":"ContainerStarted","Data":"48c00c469f3674843b4175a32f64a97132eff6be1db413ba39138c964462d37e"}
Jul 27 17:04:30.332620 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:30.332214    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e906720c-c8d1-4be6-9f5f-82bf93864fb0","Type":"ContainerStarted","Data":"79147fb3439dafde00facb8a38dba600018e93eaca30baefd32e98ff6a2c89cf"}
Jul 27 17:04:30.332620 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:30.332263    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e906720c-c8d1-4be6-9f5f-82bf93864fb0","Type":"ContainerStarted","Data":"6aaeb2f99002dffa2e91a9b1b40cfa67dfd73d1dcfc863ed83d56ef21ea57670"}
Jul 27 17:04:30.332620 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:30.332306    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e906720c-c8d1-4be6-9f5f-82bf93864fb0","Type":"ContainerStarted","Data":"6b354d77c45f03164b237152deb86312349ecbe8f48db475c7369cb1c3dc5de7"}
Jul 27 17:04:30.332620 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:30.332344    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e906720c-c8d1-4be6-9f5f-82bf93864fb0","Type":"ContainerStarted","Data":"4d2274a16a19020eb95174416942ac18824a6d76afa95cf5ea28724922743160"}
Jul 27 17:04:30.332620 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:30.332429    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e906720c-c8d1-4be6-9f5f-82bf93864fb0","Type":"ContainerStarted","Data":"e33aeacb103c8c74ef7a2f5baab7f03d7053de2a48ff55dd2255126bb76b83fe"}
Jul 27 17:04:30.359315 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:04:30.359264    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.343290043 podStartE2EDuration="2.359247713s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.95767ms" isStatefulPod=false podCreationTimestamp="2026-07-27 17:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:04:30.358159556 +0000 UTC m=+214.433424043" watchObservedRunningTime="2026-07-27 17:04:30.359247713 +0000 UTC m=+214.434512254"
Jul 27 17:05:11.641483 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.641441    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68c8965d4b-mchhr"]
Jul 27 17:05:11.646059 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.646039    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.648545 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.648519    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 17:05:11.648677 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.648519    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 17:05:11.649501 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.649473    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 17:05:11.649611 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.649589    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 17:05:11.649692 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.649612    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 17:05:11.649692 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.649626    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bg6n7\""
Jul 27 17:05:11.649692 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.649626    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 17:05:11.649692 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.649600    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 17:05:11.654291 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.654270    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 17:05:11.655013 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.654994    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68c8965d4b-mchhr"]
Jul 27 17:05:11.682653 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.682623    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee8ddad4-64b3-43a8-8692-84043ba4c045-console-oauth-config\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.682653 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.682658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-console-config\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.682873 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.682683    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee8ddad4-64b3-43a8-8692-84043ba4c045-console-serving-cert\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.682873 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.682701    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-service-ca\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.682873 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.682725    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbqk5\" (UniqueName: \"kubernetes.io/projected/ee8ddad4-64b3-43a8-8692-84043ba4c045-kube-api-access-lbqk5\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.682873 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.682790    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-trusted-ca-bundle\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.682873 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.682815    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-oauth-serving-cert\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.783735 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.783699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-trusted-ca-bundle\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.783735 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.783734    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-oauth-serving-cert\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.783987 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.783765    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee8ddad4-64b3-43a8-8692-84043ba4c045-console-oauth-config\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.783987 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.783788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-console-config\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.783987 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.783812    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee8ddad4-64b3-43a8-8692-84043ba4c045-console-serving-cert\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.783987 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.783829    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-service-ca\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.783987 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.783846    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbqk5\" (UniqueName: \"kubernetes.io/projected/ee8ddad4-64b3-43a8-8692-84043ba4c045-kube-api-access-lbqk5\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.784602 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.784570    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-oauth-serving-cert\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.784741 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.784589    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-console-config\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.784741 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.784697    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-service-ca\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.784741 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.784725    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-trusted-ca-bundle\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.786351 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.786320    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee8ddad4-64b3-43a8-8692-84043ba4c045-console-oauth-config\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.786495 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.786474    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee8ddad4-64b3-43a8-8692-84043ba4c045-console-serving-cert\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.791854 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.791831    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbqk5\" (UniqueName: \"kubernetes.io/projected/ee8ddad4-64b3-43a8-8692-84043ba4c045-kube-api-access-lbqk5\") pod \"console-68c8965d4b-mchhr\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") " pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:11.956243 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:11.956208    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:12.092385 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:12.092350    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68c8965d4b-mchhr"]
Jul 27 17:05:12.097212 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:05:12.097178    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8ddad4_64b3_43a8_8692_84043ba4c045.slice/crio-16ad1f30927c96299145e243925f98fd821cb4c330aa6e2de7e76dd548495084 WatchSource:0}: Error finding container 16ad1f30927c96299145e243925f98fd821cb4c330aa6e2de7e76dd548495084: Status 404 returned error can't find the container with id 16ad1f30927c96299145e243925f98fd821cb4c330aa6e2de7e76dd548495084
Jul 27 17:05:12.504422 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:12.504384    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c8965d4b-mchhr" event={"ID":"ee8ddad4-64b3-43a8-8692-84043ba4c045","Type":"ContainerStarted","Data":"2b51abcb5f8f924d38c48c47e80cf302e8cd7510a48df6de71dcae52c49ed1b7"}
Jul 27 17:05:12.504594 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:12.504435    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c8965d4b-mchhr" event={"ID":"ee8ddad4-64b3-43a8-8692-84043ba4c045","Type":"ContainerStarted","Data":"16ad1f30927c96299145e243925f98fd821cb4c330aa6e2de7e76dd548495084"}
Jul 27 17:05:12.527417 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:12.527361    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68c8965d4b-mchhr" podStartSLOduration=1.527344303 podStartE2EDuration="1.527344303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:05:12.52559157 +0000 UTC m=+256.600856057" watchObservedRunningTime="2026-07-27 17:05:12.527344303 +0000 UTC m=+256.602608789"
Jul 27 17:05:21.956588 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:21.956492    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:21.956588 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:21.956558    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:21.962907 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:21.962872    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:22.549420 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:22.549394    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:05:50.977715 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:50.977680    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl"]
Jul 27 17:05:50.981017 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:50.980999    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl"
Jul 27 17:05:50.988987 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:50.988963    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 27 17:05:50.989546 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:50.989520    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 27 17:05:50.989546 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:50.989535    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fmp6k\""
Jul 27 17:05:50.996563 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:50.996537    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl"]
Jul 27 17:05:51.115315 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:51.115264    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23ef067c-87b0-42de-a56c-e438893f77f4-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl\" (UID: \"23ef067c-87b0-42de-a56c-e438893f77f4\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl"
Jul 27 17:05:51.115503 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:51.115336    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg47m\" (UniqueName: \"kubernetes.io/projected/23ef067c-87b0-42de-a56c-e438893f77f4-kube-api-access-mg47m\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl\" (UID: \"23ef067c-87b0-42de-a56c-e438893f77f4\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl"
Jul 27 17:05:51.115503 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:51.115379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23ef067c-87b0-42de-a56c-e438893f77f4-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl\" (UID: \"23ef067c-87b0-42de-a56c-e438893f77f4\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl"
Jul 27 17:05:51.216147 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:51.216102    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23ef067c-87b0-42de-a56c-e438893f77f4-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl\" (UID: \"23ef067c-87b0-42de-a56c-e438893f77f4\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl"
Jul 27 17:05:51.216369 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:51.216190    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23ef067c-87b0-42de-a56c-e438893f77f4-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl\" (UID: \"23ef067c-87b0-42de-a56c-e438893f77f4\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl"
Jul 27 17:05:51.216369 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:51.216238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg47m\" (UniqueName: \"kubernetes.io/projected/23ef067c-87b0-42de-a56c-e438893f77f4-kube-api-access-mg47m\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl\" (UID: \"23ef067c-87b0-42de-a56c-e438893f77f4\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl"
Jul 27 17:05:51.216602 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:51.216579    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23ef067c-87b0-42de-a56c-e438893f77f4-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl\" (UID: \"23ef067c-87b0-42de-a56c-e438893f77f4\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl"
Jul 27 17:05:51.216602 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:51.216591    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23ef067c-87b0-42de-a56c-e438893f77f4-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl\" (UID: \"23ef067c-87b0-42de-a56c-e438893f77f4\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl"
Jul 27 17:05:51.227026 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:51.226997    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg47m\" (UniqueName: \"kubernetes.io/projected/23ef067c-87b0-42de-a56c-e438893f77f4-kube-api-access-mg47m\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl\" (UID: \"23ef067c-87b0-42de-a56c-e438893f77f4\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl"
Jul 27 17:05:51.290368 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:51.290290    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl"
Jul 27 17:05:51.418980 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:51.418950    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl"]
Jul 27 17:05:51.421450 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:05:51.421423    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ef067c_87b0_42de_a56c_e438893f77f4.slice/crio-ddf678ca1783d9cc45ed48760a59143004d7ce9394e3f3eefa30e4f40d92f656 WatchSource:0}: Error finding container ddf678ca1783d9cc45ed48760a59143004d7ce9394e3f3eefa30e4f40d92f656: Status 404 returned error can't find the container with id ddf678ca1783d9cc45ed48760a59143004d7ce9394e3f3eefa30e4f40d92f656
Jul 27 17:05:51.661670 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:51.661586    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl" event={"ID":"23ef067c-87b0-42de-a56c-e438893f77f4","Type":"ContainerStarted","Data":"ddf678ca1783d9cc45ed48760a59143004d7ce9394e3f3eefa30e4f40d92f656"}
Jul 27 17:05:57.263318 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:57.263292    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:05:57.265753 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:57.263343    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:05:57.266593 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:57.266573    2566 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 17:05:57.691108 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:57.691026    2566 generic.go:356] "Generic (PLEG): container finished" podID="23ef067c-87b0-42de-a56c-e438893f77f4" containerID="0cbfb094dd13ffa5c27f127ab003bfc09c6c732abc71b492ba5a00240b691392" exitCode=0
Jul 27 17:05:57.691272 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:57.691108    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl" event={"ID":"23ef067c-87b0-42de-a56c-e438893f77f4","Type":"ContainerDied","Data":"0cbfb094dd13ffa5c27f127ab003bfc09c6c732abc71b492ba5a00240b691392"}
Jul 27 17:05:57.692134 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:05:57.692116    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:06:00.708595 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:00.708563    2566 generic.go:356] "Generic (PLEG): container finished" podID="23ef067c-87b0-42de-a56c-e438893f77f4" containerID="b531ad1298374b684365508abf458f8824dc5b4a9c239025db52cdb578995a86" exitCode=0
Jul 27 17:06:00.708984 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:00.708636    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl" event={"ID":"23ef067c-87b0-42de-a56c-e438893f77f4","Type":"ContainerDied","Data":"b531ad1298374b684365508abf458f8824dc5b4a9c239025db52cdb578995a86"}
Jul 27 17:06:07.745130 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:07.745099    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl" event={"ID":"23ef067c-87b0-42de-a56c-e438893f77f4","Type":"ContainerStarted","Data":"8e79cda736f3a25b5a71921a6d2d6f6561ca3389988208cb135a365635647a3e"}
Jul 27 17:06:07.765788 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:07.765728    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl" podStartSLOduration=2.965449225 podStartE2EDuration="17.765703309s" totalImagesPullingTime="14.800254084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:05:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:06:07.763812276 +0000 UTC m=+311.839076763" watchObservedRunningTime="2026-07-27 17:06:07.765703309 +0000 UTC m=+311.840967796"
Jul 27 17:06:08.751341 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:08.751305    2566 generic.go:356] "Generic (PLEG): container finished" podID="23ef067c-87b0-42de-a56c-e438893f77f4" containerID="8e79cda736f3a25b5a71921a6d2d6f6561ca3389988208cb135a365635647a3e" exitCode=0
Jul 27 17:06:08.751341 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:08.751346    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl" event={"ID":"23ef067c-87b0-42de-a56c-e438893f77f4","Type":"ContainerDied","Data":"8e79cda736f3a25b5a71921a6d2d6f6561ca3389988208cb135a365635647a3e"}
Jul 27 17:06:09.879211 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:09.879188    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl"
Jul 27 17:06:09.977043 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:09.977008    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23ef067c-87b0-42de-a56c-e438893f77f4-util\") pod \"23ef067c-87b0-42de-a56c-e438893f77f4\" (UID: \"23ef067c-87b0-42de-a56c-e438893f77f4\") "
Jul 27 17:06:09.977444 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:09.977072    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23ef067c-87b0-42de-a56c-e438893f77f4-bundle\") pod \"23ef067c-87b0-42de-a56c-e438893f77f4\" (UID: \"23ef067c-87b0-42de-a56c-e438893f77f4\") "
Jul 27 17:06:09.977444 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:09.977150    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg47m\" (UniqueName: \"kubernetes.io/projected/23ef067c-87b0-42de-a56c-e438893f77f4-kube-api-access-mg47m\") pod \"23ef067c-87b0-42de-a56c-e438893f77f4\" (UID: \"23ef067c-87b0-42de-a56c-e438893f77f4\") "
Jul 27 17:06:09.977881 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:09.977849    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ef067c-87b0-42de-a56c-e438893f77f4-bundle" (OuterVolumeSpecName: "bundle") pod "23ef067c-87b0-42de-a56c-e438893f77f4" (UID: "23ef067c-87b0-42de-a56c-e438893f77f4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:06:09.979324 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:09.979295    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ef067c-87b0-42de-a56c-e438893f77f4-kube-api-access-mg47m" (OuterVolumeSpecName: "kube-api-access-mg47m") pod "23ef067c-87b0-42de-a56c-e438893f77f4" (UID: "23ef067c-87b0-42de-a56c-e438893f77f4"). InnerVolumeSpecName "kube-api-access-mg47m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:06:09.982147 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:09.982120    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ef067c-87b0-42de-a56c-e438893f77f4-util" (OuterVolumeSpecName: "util") pod "23ef067c-87b0-42de-a56c-e438893f77f4" (UID: "23ef067c-87b0-42de-a56c-e438893f77f4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:06:10.077979 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:10.077882    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23ef067c-87b0-42de-a56c-e438893f77f4-util\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:06:10.077979 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:10.077917    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23ef067c-87b0-42de-a56c-e438893f77f4-bundle\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:06:10.077979 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:10.077930    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg47m\" (UniqueName: \"kubernetes.io/projected/23ef067c-87b0-42de-a56c-e438893f77f4-kube-api-access-mg47m\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:06:10.762205 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:10.762160    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl"
Jul 27 17:06:10.762366 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:10.762154    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xxptl" event={"ID":"23ef067c-87b0-42de-a56c-e438893f77f4","Type":"ContainerDied","Data":"ddf678ca1783d9cc45ed48760a59143004d7ce9394e3f3eefa30e4f40d92f656"}
Jul 27 17:06:10.762366 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:10.762304    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddf678ca1783d9cc45ed48760a59143004d7ce9394e3f3eefa30e4f40d92f656"
Jul 27 17:06:13.559345 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.559303    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28"]
Jul 27 17:06:13.559822 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.559782    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ef067c-87b0-42de-a56c-e438893f77f4" containerName="pull"
Jul 27 17:06:13.559822 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.559801    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ef067c-87b0-42de-a56c-e438893f77f4" containerName="pull"
Jul 27 17:06:13.559822 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.559815    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ef067c-87b0-42de-a56c-e438893f77f4" containerName="util"
Jul 27 17:06:13.559822 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.559823    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ef067c-87b0-42de-a56c-e438893f77f4" containerName="util"
Jul 27 17:06:13.560020 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.559849    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ef067c-87b0-42de-a56c-e438893f77f4" containerName="extract"
Jul 27 17:06:13.560020 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.559859    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ef067c-87b0-42de-a56c-e438893f77f4" containerName="extract"
Jul 27 17:06:13.560020 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.559949    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ef067c-87b0-42de-a56c-e438893f77f4" containerName="extract"
Jul 27 17:06:13.595917 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.595888    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28"]
Jul 27 17:06:13.596066 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.595997    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28"
Jul 27 17:06:13.598883 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.598857    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 27 17:06:13.599054 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.599037    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-djc8v\""
Jul 27 17:06:13.599476 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.599461    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 27 17:06:13.600338 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.600321    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 27 17:06:13.710513 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.710472    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn9tz\" (UniqueName: \"kubernetes.io/projected/51a5c4f1-8809-446a-90e5-a987d68ffb14-kube-api-access-wn9tz\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-d7z28\" (UID: \"51a5c4f1-8809-446a-90e5-a987d68ffb14\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28"
Jul 27 17:06:13.710689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.710542    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/51a5c4f1-8809-446a-90e5-a987d68ffb14-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-d7z28\" (UID: \"51a5c4f1-8809-446a-90e5-a987d68ffb14\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28"
Jul 27 17:06:13.811642 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.811547    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn9tz\" (UniqueName: \"kubernetes.io/projected/51a5c4f1-8809-446a-90e5-a987d68ffb14-kube-api-access-wn9tz\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-d7z28\" (UID: \"51a5c4f1-8809-446a-90e5-a987d68ffb14\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28"
Jul 27 17:06:13.811642 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.811618    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/51a5c4f1-8809-446a-90e5-a987d68ffb14-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-d7z28\" (UID: \"51a5c4f1-8809-446a-90e5-a987d68ffb14\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28"
Jul 27 17:06:13.813998 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.813965    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/51a5c4f1-8809-446a-90e5-a987d68ffb14-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-d7z28\" (UID: \"51a5c4f1-8809-446a-90e5-a987d68ffb14\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28"
Jul 27 17:06:13.826029 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.826002    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn9tz\" (UniqueName: \"kubernetes.io/projected/51a5c4f1-8809-446a-90e5-a987d68ffb14-kube-api-access-wn9tz\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-d7z28\" (UID: \"51a5c4f1-8809-446a-90e5-a987d68ffb14\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28"
Jul 27 17:06:13.906501 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:13.906466    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28"
Jul 27 17:06:14.059540 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:14.059515    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28"]
Jul 27 17:06:14.061418 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:06:14.061389    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a5c4f1_8809_446a_90e5_a987d68ffb14.slice/crio-e034206dd7aabebc92fd522b329f5055dda9fd6a2eaf3cc9a6535ebb027b7af3 WatchSource:0}: Error finding container e034206dd7aabebc92fd522b329f5055dda9fd6a2eaf3cc9a6535ebb027b7af3: Status 404 returned error can't find the container with id e034206dd7aabebc92fd522b329f5055dda9fd6a2eaf3cc9a6535ebb027b7af3
Jul 27 17:06:14.779025 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:14.778986    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28" event={"ID":"51a5c4f1-8809-446a-90e5-a987d68ffb14","Type":"ContainerStarted","Data":"e034206dd7aabebc92fd522b329f5055dda9fd6a2eaf3cc9a6535ebb027b7af3"}
Jul 27 17:06:17.800921 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:17.800828    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28" event={"ID":"51a5c4f1-8809-446a-90e5-a987d68ffb14","Type":"ContainerStarted","Data":"a040a617960e1d020c92d028caf985a1690fe70ec27c228ab7d37d522888210e"}
Jul 27 17:06:17.801288 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:17.800981    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28"
Jul 27 17:06:17.821539 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:17.821489    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28" podStartSLOduration=1.4138523539999999 podStartE2EDuration="4.821471718s" totalImagesPullingTime="3.407619364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:06:17.819728167 +0000 UTC m=+321.894992655" watchObservedRunningTime="2026-07-27 17:06:17.821471718 +0000 UTC m=+321.896736205"
Jul 27 17:06:18.304049 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:18.304013    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-7c8hf"]
Jul 27 17:06:18.319845 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:18.319817    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-7c8hf"
Jul 27 17:06:18.320017 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:18.319964    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-7c8hf"]
Jul 27 17:06:18.322338 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:18.322310    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 27 17:06:18.322455 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:18.322336    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 27 17:06:18.322455 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:18.322372    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-6564d\""
Jul 27 17:06:18.457459 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:18.457424    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdw7f\" (UniqueName: \"kubernetes.io/projected/68dcdaf5-0f80-4073-9f22-b69cc68d015d-kube-api-access-fdw7f\") pod \"keda-operator-77f648d694-7c8hf\" (UID: \"68dcdaf5-0f80-4073-9f22-b69cc68d015d\") " pod="openshift-keda/keda-operator-77f648d694-7c8hf"
Jul 27 17:06:18.457666 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:18.457560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/68dcdaf5-0f80-4073-9f22-b69cc68d015d-cabundle0\") pod \"keda-operator-77f648d694-7c8hf\" (UID: \"68dcdaf5-0f80-4073-9f22-b69cc68d015d\") " pod="openshift-keda/keda-operator-77f648d694-7c8hf"
Jul 27 17:06:18.457666 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:18.457643    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/68dcdaf5-0f80-4073-9f22-b69cc68d015d-certificates\") pod \"keda-operator-77f648d694-7c8hf\" (UID: \"68dcdaf5-0f80-4073-9f22-b69cc68d015d\") " pod="openshift-keda/keda-operator-77f648d694-7c8hf"
Jul 27 17:06:18.558192 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:18.558092    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/68dcdaf5-0f80-4073-9f22-b69cc68d015d-cabundle0\") pod \"keda-operator-77f648d694-7c8hf\" (UID: \"68dcdaf5-0f80-4073-9f22-b69cc68d015d\") " pod="openshift-keda/keda-operator-77f648d694-7c8hf"
Jul 27 17:06:18.558192 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:18.558146    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/68dcdaf5-0f80-4073-9f22-b69cc68d015d-certificates\") pod \"keda-operator-77f648d694-7c8hf\" (UID: \"68dcdaf5-0f80-4073-9f22-b69cc68d015d\") " pod="openshift-keda/keda-operator-77f648d694-7c8hf"
Jul 27 17:06:18.558192 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:18.558188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdw7f\" (UniqueName: \"kubernetes.io/projected/68dcdaf5-0f80-4073-9f22-b69cc68d015d-kube-api-access-fdw7f\") pod \"keda-operator-77f648d694-7c8hf\" (UID: \"68dcdaf5-0f80-4073-9f22-b69cc68d015d\") " pod="openshift-keda/keda-operator-77f648d694-7c8hf"
Jul 27 17:06:18.558467 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:06:18.558274    2566 secret.go:281] references non-existent secret key: ca.crt
Jul 27 17:06:18.558467 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:06:18.558297    2566 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 27 17:06:18.558467 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:06:18.558309    2566 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-7c8hf: references non-existent secret key: ca.crt
Jul 27 17:06:18.558467 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:06:18.558382    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68dcdaf5-0f80-4073-9f22-b69cc68d015d-certificates podName:68dcdaf5-0f80-4073-9f22-b69cc68d015d nodeName:}" failed. No retries permitted until 2026-07-27 17:06:19.058360393 +0000 UTC m=+323.133624860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/68dcdaf5-0f80-4073-9f22-b69cc68d015d-certificates") pod "keda-operator-77f648d694-7c8hf" (UID: "68dcdaf5-0f80-4073-9f22-b69cc68d015d") : references non-existent secret key: ca.crt
Jul 27 17:06:18.558823 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:18.558805    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/68dcdaf5-0f80-4073-9f22-b69cc68d015d-cabundle0\") pod \"keda-operator-77f648d694-7c8hf\" (UID: \"68dcdaf5-0f80-4073-9f22-b69cc68d015d\") " pod="openshift-keda/keda-operator-77f648d694-7c8hf"
Jul 27 17:06:18.566376 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:18.566355    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdw7f\" (UniqueName: \"kubernetes.io/projected/68dcdaf5-0f80-4073-9f22-b69cc68d015d-kube-api-access-fdw7f\") pod \"keda-operator-77f648d694-7c8hf\" (UID: \"68dcdaf5-0f80-4073-9f22-b69cc68d015d\") " pod="openshift-keda/keda-operator-77f648d694-7c8hf"
Jul 27 17:06:19.063468 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:19.063427    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/68dcdaf5-0f80-4073-9f22-b69cc68d015d-certificates\") pod \"keda-operator-77f648d694-7c8hf\" (UID: \"68dcdaf5-0f80-4073-9f22-b69cc68d015d\") " pod="openshift-keda/keda-operator-77f648d694-7c8hf"
Jul 27 17:06:19.066264 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:19.066231    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/68dcdaf5-0f80-4073-9f22-b69cc68d015d-certificates\") pod \"keda-operator-77f648d694-7c8hf\" (UID: \"68dcdaf5-0f80-4073-9f22-b69cc68d015d\") " pod="openshift-keda/keda-operator-77f648d694-7c8hf"
Jul 27 17:06:19.231583 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:19.231546    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-7c8hf"
Jul 27 17:06:19.377783 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:19.377757    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-7c8hf"]
Jul 27 17:06:19.381345 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:06:19.381302    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68dcdaf5_0f80_4073_9f22_b69cc68d015d.slice/crio-f1bb0875a43f0b266256b4b172897bb3e2ed27d84f3a6bd351784cff90f60cb5 WatchSource:0}: Error finding container f1bb0875a43f0b266256b4b172897bb3e2ed27d84f3a6bd351784cff90f60cb5: Status 404 returned error can't find the container with id f1bb0875a43f0b266256b4b172897bb3e2ed27d84f3a6bd351784cff90f60cb5
Jul 27 17:06:19.810495 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:19.810462    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-7c8hf" event={"ID":"68dcdaf5-0f80-4073-9f22-b69cc68d015d","Type":"ContainerStarted","Data":"f1bb0875a43f0b266256b4b172897bb3e2ed27d84f3a6bd351784cff90f60cb5"}
Jul 27 17:06:22.829048 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:22.829011    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-7c8hf" event={"ID":"68dcdaf5-0f80-4073-9f22-b69cc68d015d","Type":"ContainerStarted","Data":"518679cac62535508906b8173a2c934abc7273ef8e158aeab1fe43882e066169"}
Jul 27 17:06:22.829460 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:22.829083    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-7c8hf"
Jul 27 17:06:22.847265 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:22.846736    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-7c8hf" podStartSLOduration=1.90373032 podStartE2EDuration="4.846716854s" totalImagesPullingTime="2.942986534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:06:22.845958656 +0000 UTC m=+326.921223141" watchObservedRunningTime="2026-07-27 17:06:22.846716854 +0000 UTC m=+326.921981343"
Jul 27 17:06:38.807251 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:38.807219    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28"
Jul 27 17:06:43.835411 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:06:43.835383    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-7c8hf"
Jul 27 17:07:37.538039 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.538000    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx"]
Jul 27 17:07:37.541379 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.541354    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx"
Jul 27 17:07:37.544180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.544147    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 27 17:07:37.544291 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.544181    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 27 17:07:37.544291 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.544229    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-vmv6v\""
Jul 27 17:07:37.545154 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.545134    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 27 17:07:37.550656 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.550627    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx"]
Jul 27 17:07:37.574735 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.570525    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw"]
Jul 27 17:07:37.575338 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.575319    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw"
Jul 27 17:07:37.579151 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.579131    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 27 17:07:37.579289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.579272    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-kx8cq\""
Jul 27 17:07:37.584311 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.584289    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw"]
Jul 27 17:07:37.716815 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.716777    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq57m\" (UniqueName: \"kubernetes.io/projected/57f1804c-717f-46a4-8b1c-8ddfd1af84e6-kube-api-access-mq57m\") pod \"kserve-controller-manager-6ccfdc6bb8-mzzcx\" (UID: \"57f1804c-717f-46a4-8b1c-8ddfd1af84e6\") " pod="kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx"
Jul 27 17:07:37.716815 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.716820    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32959584-b9bb-4d73-92ad-5433f7c5f44b-cert\") pod \"llmisvc-controller-manager-77cb7f7d68-fl4nw\" (UID: \"32959584-b9bb-4d73-92ad-5433f7c5f44b\") " pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw"
Jul 27 17:07:37.717015 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.716912    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57f1804c-717f-46a4-8b1c-8ddfd1af84e6-cert\") pod \"kserve-controller-manager-6ccfdc6bb8-mzzcx\" (UID: \"57f1804c-717f-46a4-8b1c-8ddfd1af84e6\") " pod="kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx"
Jul 27 17:07:37.717015 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.716985    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94ltd\" (UniqueName: \"kubernetes.io/projected/32959584-b9bb-4d73-92ad-5433f7c5f44b-kube-api-access-94ltd\") pod \"llmisvc-controller-manager-77cb7f7d68-fl4nw\" (UID: \"32959584-b9bb-4d73-92ad-5433f7c5f44b\") " pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw"
Jul 27 17:07:37.817706 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.817612    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57f1804c-717f-46a4-8b1c-8ddfd1af84e6-cert\") pod \"kserve-controller-manager-6ccfdc6bb8-mzzcx\" (UID: \"57f1804c-717f-46a4-8b1c-8ddfd1af84e6\") " pod="kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx"
Jul 27 17:07:37.817706 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.817693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94ltd\" (UniqueName: \"kubernetes.io/projected/32959584-b9bb-4d73-92ad-5433f7c5f44b-kube-api-access-94ltd\") pod \"llmisvc-controller-manager-77cb7f7d68-fl4nw\" (UID: \"32959584-b9bb-4d73-92ad-5433f7c5f44b\") " pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw"
Jul 27 17:07:37.817890 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.817745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq57m\" (UniqueName: \"kubernetes.io/projected/57f1804c-717f-46a4-8b1c-8ddfd1af84e6-kube-api-access-mq57m\") pod \"kserve-controller-manager-6ccfdc6bb8-mzzcx\" (UID: \"57f1804c-717f-46a4-8b1c-8ddfd1af84e6\") " pod="kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx"
Jul 27 17:07:37.817890 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.817774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32959584-b9bb-4d73-92ad-5433f7c5f44b-cert\") pod \"llmisvc-controller-manager-77cb7f7d68-fl4nw\" (UID: \"32959584-b9bb-4d73-92ad-5433f7c5f44b\") " pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw"
Jul 27 17:07:37.820190 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.820150    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57f1804c-717f-46a4-8b1c-8ddfd1af84e6-cert\") pod \"kserve-controller-manager-6ccfdc6bb8-mzzcx\" (UID: \"57f1804c-717f-46a4-8b1c-8ddfd1af84e6\") " pod="kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx"
Jul 27 17:07:37.820316 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.820221    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32959584-b9bb-4d73-92ad-5433f7c5f44b-cert\") pod \"llmisvc-controller-manager-77cb7f7d68-fl4nw\" (UID: \"32959584-b9bb-4d73-92ad-5433f7c5f44b\") " pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw"
Jul 27 17:07:37.825936 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.825910    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94ltd\" (UniqueName: \"kubernetes.io/projected/32959584-b9bb-4d73-92ad-5433f7c5f44b-kube-api-access-94ltd\") pod \"llmisvc-controller-manager-77cb7f7d68-fl4nw\" (UID: \"32959584-b9bb-4d73-92ad-5433f7c5f44b\") " pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw"
Jul 27 17:07:37.826129 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.826111    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq57m\" (UniqueName: \"kubernetes.io/projected/57f1804c-717f-46a4-8b1c-8ddfd1af84e6-kube-api-access-mq57m\") pod \"kserve-controller-manager-6ccfdc6bb8-mzzcx\" (UID: \"57f1804c-717f-46a4-8b1c-8ddfd1af84e6\") " pod="kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx"
Jul 27 17:07:37.851880 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.851839    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx"
Jul 27 17:07:37.885756 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.885727    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw"
Jul 27 17:07:37.991372 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:37.991341    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx"]
Jul 27 17:07:37.993515 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:07:37.993487    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f1804c_717f_46a4_8b1c_8ddfd1af84e6.slice/crio-19842464aa8c84acd35b3d1cdf04c7e07e99a1900a6076459c84b9966baf6363 WatchSource:0}: Error finding container 19842464aa8c84acd35b3d1cdf04c7e07e99a1900a6076459c84b9966baf6363: Status 404 returned error can't find the container with id 19842464aa8c84acd35b3d1cdf04c7e07e99a1900a6076459c84b9966baf6363
Jul 27 17:07:38.023983 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:38.023950    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw"]
Jul 27 17:07:38.026005 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:07:38.025977    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod32959584_b9bb_4d73_92ad_5433f7c5f44b.slice/crio-fa0c74659ac2c5b1859af449e304ddbc5b7b26e33313fd38002f4866ea2af6f1 WatchSource:0}: Error finding container fa0c74659ac2c5b1859af449e304ddbc5b7b26e33313fd38002f4866ea2af6f1: Status 404 returned error can't find the container with id fa0c74659ac2c5b1859af449e304ddbc5b7b26e33313fd38002f4866ea2af6f1
Jul 27 17:07:38.154744 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:38.154648    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw" event={"ID":"32959584-b9bb-4d73-92ad-5433f7c5f44b","Type":"ContainerStarted","Data":"fa0c74659ac2c5b1859af449e304ddbc5b7b26e33313fd38002f4866ea2af6f1"}
Jul 27 17:07:38.155758 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:38.155734    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx" event={"ID":"57f1804c-717f-46a4-8b1c-8ddfd1af84e6","Type":"ContainerStarted","Data":"19842464aa8c84acd35b3d1cdf04c7e07e99a1900a6076459c84b9966baf6363"}
Jul 27 17:07:42.177118 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:42.177023    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw" event={"ID":"32959584-b9bb-4d73-92ad-5433f7c5f44b","Type":"ContainerStarted","Data":"4b4d3b820cbd1ca5714a22dc7bf2e4de5a6b29e15a1fa7600fc61b4872973286"}
Jul 27 17:07:42.177566 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:42.177133    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw"
Jul 27 17:07:42.178555 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:42.178530    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx" event={"ID":"57f1804c-717f-46a4-8b1c-8ddfd1af84e6","Type":"ContainerStarted","Data":"af5b940e57fa39c1d54541ca7f340434cbd5c168eb1b5717c4c418c6b69dfb32"}
Jul 27 17:07:42.178684 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:42.178570    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx"
Jul 27 17:07:42.194941 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:42.194899    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw" podStartSLOduration=1.493585656 podStartE2EDuration="5.194885714s" totalImagesPullingTime="3.701300058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:07:42.193709689 +0000 UTC m=+406.268974177" watchObservedRunningTime="2026-07-27 17:07:42.194885714 +0000 UTC m=+406.270150179"
Jul 27 17:07:42.209730 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:07:42.209673    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx" podStartSLOduration=1.475970025 podStartE2EDuration="5.20965898s" totalImagesPullingTime="3.733688955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:07:42.208843921 +0000 UTC m=+406.284108401" watchObservedRunningTime="2026-07-27 17:07:42.20965898 +0000 UTC m=+406.284923467"
Jul 27 17:08:13.184878 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:13.184845    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw"
Jul 27 17:08:13.187986 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:13.187961    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx"
Jul 27 17:08:20.671270 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:20.671179    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc"]
Jul 27 17:08:20.674905 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:20.674881    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc"
Jul 27 17:08:20.677410 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:20.677389    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 27 17:08:20.677683 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:20.677658    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 27 17:08:20.678347 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:20.678332    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fmp6k\""
Jul 27 17:08:20.682862 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:20.682843    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc"]
Jul 27 17:08:20.792035 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:20.792000    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3fe4494-301d-463a-807b-95087909d772-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc\" (UID: \"a3fe4494-301d-463a-807b-95087909d772\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc"
Jul 27 17:08:20.792035 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:20.792034    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3fe4494-301d-463a-807b-95087909d772-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc\" (UID: \"a3fe4494-301d-463a-807b-95087909d772\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc"
Jul 27 17:08:20.792261 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:20.792056    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlgr8\" (UniqueName: \"kubernetes.io/projected/a3fe4494-301d-463a-807b-95087909d772-kube-api-access-zlgr8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc\" (UID: \"a3fe4494-301d-463a-807b-95087909d772\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc"
Jul 27 17:08:20.893185 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:20.893127    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3fe4494-301d-463a-807b-95087909d772-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc\" (UID: \"a3fe4494-301d-463a-807b-95087909d772\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc"
Jul 27 17:08:20.893360 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:20.893226    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3fe4494-301d-463a-807b-95087909d772-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc\" (UID: \"a3fe4494-301d-463a-807b-95087909d772\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc"
Jul 27 17:08:20.893360 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:20.893247    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlgr8\" (UniqueName: \"kubernetes.io/projected/a3fe4494-301d-463a-807b-95087909d772-kube-api-access-zlgr8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc\" (UID: \"a3fe4494-301d-463a-807b-95087909d772\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc"
Jul 27 17:08:20.893564 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:20.893542    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3fe4494-301d-463a-807b-95087909d772-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc\" (UID: \"a3fe4494-301d-463a-807b-95087909d772\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc"
Jul 27 17:08:20.893604 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:20.893558    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3fe4494-301d-463a-807b-95087909d772-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc\" (UID: \"a3fe4494-301d-463a-807b-95087909d772\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc"
Jul 27 17:08:20.901684 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:20.901649    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlgr8\" (UniqueName: \"kubernetes.io/projected/a3fe4494-301d-463a-807b-95087909d772-kube-api-access-zlgr8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc\" (UID: \"a3fe4494-301d-463a-807b-95087909d772\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc"
Jul 27 17:08:20.985271 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:20.985237    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc"
Jul 27 17:08:21.115448 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:21.115387    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc"]
Jul 27 17:08:21.117619 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:08:21.117592    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3fe4494_301d_463a_807b_95087909d772.slice/crio-1159bd7892c00b432bbc74cfceeae4c9648f8270afdb6c4450c8da6ecada699a WatchSource:0}: Error finding container 1159bd7892c00b432bbc74cfceeae4c9648f8270afdb6c4450c8da6ecada699a: Status 404 returned error can't find the container with id 1159bd7892c00b432bbc74cfceeae4c9648f8270afdb6c4450c8da6ecada699a
Jul 27 17:08:21.355139 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:21.355050    2566 generic.go:356] "Generic (PLEG): container finished" podID="a3fe4494-301d-463a-807b-95087909d772" containerID="9615bb25f9df03b2473d4bc132110c2478ee4efb3cfcbfea3386a2e04d703a8a" exitCode=0
Jul 27 17:08:21.355316 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:21.355145    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc" event={"ID":"a3fe4494-301d-463a-807b-95087909d772","Type":"ContainerDied","Data":"9615bb25f9df03b2473d4bc132110c2478ee4efb3cfcbfea3386a2e04d703a8a"}
Jul 27 17:08:21.355316 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:21.355217    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc" event={"ID":"a3fe4494-301d-463a-807b-95087909d772","Type":"ContainerStarted","Data":"1159bd7892c00b432bbc74cfceeae4c9648f8270afdb6c4450c8da6ecada699a"}
Jul 27 17:08:24.371977 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:24.371942    2566 generic.go:356] "Generic (PLEG): container finished" podID="a3fe4494-301d-463a-807b-95087909d772" containerID="2092d87f6216ec152d9460c445c04ad5d1eb3996be785eac745ca4dd45d061e7" exitCode=0
Jul 27 17:08:24.372352 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:24.372005    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc" event={"ID":"a3fe4494-301d-463a-807b-95087909d772","Type":"ContainerDied","Data":"2092d87f6216ec152d9460c445c04ad5d1eb3996be785eac745ca4dd45d061e7"}
Jul 27 17:08:25.379037 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:25.379000    2566 generic.go:356] "Generic (PLEG): container finished" podID="a3fe4494-301d-463a-807b-95087909d772" containerID="359ee2bb0522bc3db474e6fbc638efd79043eb3d50c0f021c34e4f5fbd97f543" exitCode=0
Jul 27 17:08:25.379429 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:25.379088    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc" event={"ID":"a3fe4494-301d-463a-807b-95087909d772","Type":"ContainerDied","Data":"359ee2bb0522bc3db474e6fbc638efd79043eb3d50c0f021c34e4f5fbd97f543"}
Jul 27 17:08:26.513418 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:26.513392    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc"
Jul 27 17:08:26.645375 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:26.645291    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3fe4494-301d-463a-807b-95087909d772-util\") pod \"a3fe4494-301d-463a-807b-95087909d772\" (UID: \"a3fe4494-301d-463a-807b-95087909d772\") "
Jul 27 17:08:26.645375 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:26.645362    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3fe4494-301d-463a-807b-95087909d772-bundle\") pod \"a3fe4494-301d-463a-807b-95087909d772\" (UID: \"a3fe4494-301d-463a-807b-95087909d772\") "
Jul 27 17:08:26.645559 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:26.645434    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlgr8\" (UniqueName: \"kubernetes.io/projected/a3fe4494-301d-463a-807b-95087909d772-kube-api-access-zlgr8\") pod \"a3fe4494-301d-463a-807b-95087909d772\" (UID: \"a3fe4494-301d-463a-807b-95087909d772\") "
Jul 27 17:08:26.646210 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:26.646148    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fe4494-301d-463a-807b-95087909d772-bundle" (OuterVolumeSpecName: "bundle") pod "a3fe4494-301d-463a-807b-95087909d772" (UID: "a3fe4494-301d-463a-807b-95087909d772"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:08:26.647412 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:26.647387    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3fe4494-301d-463a-807b-95087909d772-kube-api-access-zlgr8" (OuterVolumeSpecName: "kube-api-access-zlgr8") pod "a3fe4494-301d-463a-807b-95087909d772" (UID: "a3fe4494-301d-463a-807b-95087909d772"). InnerVolumeSpecName "kube-api-access-zlgr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:08:26.650704 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:26.650682    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fe4494-301d-463a-807b-95087909d772-util" (OuterVolumeSpecName: "util") pod "a3fe4494-301d-463a-807b-95087909d772" (UID: "a3fe4494-301d-463a-807b-95087909d772"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:08:26.746548 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:26.746511    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlgr8\" (UniqueName: \"kubernetes.io/projected/a3fe4494-301d-463a-807b-95087909d772-kube-api-access-zlgr8\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:08:26.746548 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:26.746541    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3fe4494-301d-463a-807b-95087909d772-util\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:08:26.746548 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:26.746549    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3fe4494-301d-463a-807b-95087909d772-bundle\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:08:27.395057 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:27.395027    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc"
Jul 27 17:08:27.395057 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:27.395033    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hsvdc" event={"ID":"a3fe4494-301d-463a-807b-95087909d772","Type":"ContainerDied","Data":"1159bd7892c00b432bbc74cfceeae4c9648f8270afdb6c4450c8da6ecada699a"}
Jul 27 17:08:27.395285 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:27.395097    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1159bd7892c00b432bbc74cfceeae4c9648f8270afdb6c4450c8da6ecada699a"
Jul 27 17:08:33.521239 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.521204    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-bfkr4"]
Jul 27 17:08:33.521644 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.521529    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3fe4494-301d-463a-807b-95087909d772" containerName="extract"
Jul 27 17:08:33.521644 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.521539    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3fe4494-301d-463a-807b-95087909d772" containerName="extract"
Jul 27 17:08:33.521644 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.521562    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3fe4494-301d-463a-807b-95087909d772" containerName="pull"
Jul 27 17:08:33.521644 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.521568    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3fe4494-301d-463a-807b-95087909d772" containerName="pull"
Jul 27 17:08:33.521644 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.521576    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3fe4494-301d-463a-807b-95087909d772" containerName="util"
Jul 27 17:08:33.521644 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.521582    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3fe4494-301d-463a-807b-95087909d772" containerName="util"
Jul 27 17:08:33.521644 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.521638    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3fe4494-301d-463a-807b-95087909d772" containerName="extract"
Jul 27 17:08:33.525993 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.525976    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-bfkr4"
Jul 27 17:08:33.528764 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.528744    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 27 17:08:33.534386 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.531143    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-6j548\""
Jul 27 17:08:33.534386 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.531426    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:08:33.537829 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.537801    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-bfkr4"]
Jul 27 17:08:33.592420 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.592389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c084c832-675f-457f-ab72-4651edceebf4-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-bfkr4\" (UID: \"c084c832-675f-457f-ab72-4651edceebf4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-bfkr4"
Jul 27 17:08:33.592568 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.592461    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c96f\" (UniqueName: \"kubernetes.io/projected/c084c832-675f-457f-ab72-4651edceebf4-kube-api-access-2c96f\") pod \"cert-manager-operator-controller-manager-684f79c9b9-bfkr4\" (UID: \"c084c832-675f-457f-ab72-4651edceebf4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-bfkr4"
Jul 27 17:08:33.693011 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.692973    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c084c832-675f-457f-ab72-4651edceebf4-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-bfkr4\" (UID: \"c084c832-675f-457f-ab72-4651edceebf4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-bfkr4"
Jul 27 17:08:33.693251 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.693036    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c96f\" (UniqueName: \"kubernetes.io/projected/c084c832-675f-457f-ab72-4651edceebf4-kube-api-access-2c96f\") pod \"cert-manager-operator-controller-manager-684f79c9b9-bfkr4\" (UID: \"c084c832-675f-457f-ab72-4651edceebf4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-bfkr4"
Jul 27 17:08:33.693407 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.693387    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c084c832-675f-457f-ab72-4651edceebf4-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-bfkr4\" (UID: \"c084c832-675f-457f-ab72-4651edceebf4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-bfkr4"
Jul 27 17:08:33.701509 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.701487    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c96f\" (UniqueName: \"kubernetes.io/projected/c084c832-675f-457f-ab72-4651edceebf4-kube-api-access-2c96f\") pod \"cert-manager-operator-controller-manager-684f79c9b9-bfkr4\" (UID: \"c084c832-675f-457f-ab72-4651edceebf4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-bfkr4"
Jul 27 17:08:33.842329 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.842246    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-bfkr4"
Jul 27 17:08:33.995731 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:33.995704    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-bfkr4"]
Jul 27 17:08:33.997931 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:08:33.997905    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc084c832_675f_457f_ab72_4651edceebf4.slice/crio-b40f753dd2833106ca394c717f025a53612721ec643eb884ed094d4c8865f710 WatchSource:0}: Error finding container b40f753dd2833106ca394c717f025a53612721ec643eb884ed094d4c8865f710: Status 404 returned error can't find the container with id b40f753dd2833106ca394c717f025a53612721ec643eb884ed094d4c8865f710
Jul 27 17:08:34.425897 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:34.425862    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-bfkr4" event={"ID":"c084c832-675f-457f-ab72-4651edceebf4","Type":"ContainerStarted","Data":"b40f753dd2833106ca394c717f025a53612721ec643eb884ed094d4c8865f710"}
Jul 27 17:08:37.442349 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:37.442259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-bfkr4" event={"ID":"c084c832-675f-457f-ab72-4651edceebf4","Type":"ContainerStarted","Data":"4786c974243fc93518e3597dc39dbba1a159c1b644598e9eb1baa6735a032fc8"}
Jul 27 17:08:37.463415 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:37.463348    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-bfkr4" podStartSLOduration=1.446399453 podStartE2EDuration="4.463326645s" totalImagesPullingTime="3.016927192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:08:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:08:37.462758467 +0000 UTC m=+461.538022954" watchObservedRunningTime="2026-07-27 17:08:37.463326645 +0000 UTC m=+461.538591146"
Jul 27 17:08:37.508303 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:37.508267    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68c8965d4b-mchhr"]
Jul 27 17:08:42.784129 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:42.784089    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q"]
Jul 27 17:08:42.787827 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:42.787810    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q"
Jul 27 17:08:42.790373 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:42.790340    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 27 17:08:42.790373 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:42.790367    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 27 17:08:42.790583 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:42.790459    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fmp6k\""
Jul 27 17:08:42.796343 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:42.796315    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q"]
Jul 27 17:08:42.873769 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:42.873729    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4024487b-1e9f-477b-8eda-a1acfb2694e6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q\" (UID: \"4024487b-1e9f-477b-8eda-a1acfb2694e6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q"
Jul 27 17:08:42.873934 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:42.873773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkttr\" (UniqueName: \"kubernetes.io/projected/4024487b-1e9f-477b-8eda-a1acfb2694e6-kube-api-access-rkttr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q\" (UID: \"4024487b-1e9f-477b-8eda-a1acfb2694e6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q"
Jul 27 17:08:42.873934 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:42.873796    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4024487b-1e9f-477b-8eda-a1acfb2694e6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q\" (UID: \"4024487b-1e9f-477b-8eda-a1acfb2694e6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q"
Jul 27 17:08:42.975062 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:42.975021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4024487b-1e9f-477b-8eda-a1acfb2694e6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q\" (UID: \"4024487b-1e9f-477b-8eda-a1acfb2694e6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q"
Jul 27 17:08:42.975062 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:42.975065    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkttr\" (UniqueName: \"kubernetes.io/projected/4024487b-1e9f-477b-8eda-a1acfb2694e6-kube-api-access-rkttr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q\" (UID: \"4024487b-1e9f-477b-8eda-a1acfb2694e6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q"
Jul 27 17:08:42.975309 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:42.975100    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4024487b-1e9f-477b-8eda-a1acfb2694e6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q\" (UID: \"4024487b-1e9f-477b-8eda-a1acfb2694e6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q"
Jul 27 17:08:42.975443 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:42.975425    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4024487b-1e9f-477b-8eda-a1acfb2694e6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q\" (UID: \"4024487b-1e9f-477b-8eda-a1acfb2694e6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q"
Jul 27 17:08:42.975515 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:42.975496    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4024487b-1e9f-477b-8eda-a1acfb2694e6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q\" (UID: \"4024487b-1e9f-477b-8eda-a1acfb2694e6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q"
Jul 27 17:08:42.983876 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:42.983854    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkttr\" (UniqueName: \"kubernetes.io/projected/4024487b-1e9f-477b-8eda-a1acfb2694e6-kube-api-access-rkttr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q\" (UID: \"4024487b-1e9f-477b-8eda-a1acfb2694e6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q"
Jul 27 17:08:43.098227 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:43.098117    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q"
Jul 27 17:08:43.235025 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:43.234997    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q"]
Jul 27 17:08:43.237280 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:08:43.237246    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4024487b_1e9f_477b_8eda_a1acfb2694e6.slice/crio-f31cfc03f43514ea96f707a711f8e76e26b651fcdfc96c102c82d5c39a159920 WatchSource:0}: Error finding container f31cfc03f43514ea96f707a711f8e76e26b651fcdfc96c102c82d5c39a159920: Status 404 returned error can't find the container with id f31cfc03f43514ea96f707a711f8e76e26b651fcdfc96c102c82d5c39a159920
Jul 27 17:08:43.374379 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:08:43.374291    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4024487b_1e9f_477b_8eda_a1acfb2694e6.slice/crio-21b21f0e70633ca1a4574d5312e17f9a139294d1c6332569f26ada0781601dc8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4024487b_1e9f_477b_8eda_a1acfb2694e6.slice/crio-conmon-21b21f0e70633ca1a4574d5312e17f9a139294d1c6332569f26ada0781601dc8.scope\": RecentStats: unable to find data in memory cache]"
Jul 27 17:08:43.472127 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:43.472090    2566 generic.go:356] "Generic (PLEG): container finished" podID="4024487b-1e9f-477b-8eda-a1acfb2694e6" containerID="21b21f0e70633ca1a4574d5312e17f9a139294d1c6332569f26ada0781601dc8" exitCode=0
Jul 27 17:08:43.472289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:43.472124    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q" event={"ID":"4024487b-1e9f-477b-8eda-a1acfb2694e6","Type":"ContainerDied","Data":"21b21f0e70633ca1a4574d5312e17f9a139294d1c6332569f26ada0781601dc8"}
Jul 27 17:08:43.472289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:43.472194    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q" event={"ID":"4024487b-1e9f-477b-8eda-a1acfb2694e6","Type":"ContainerStarted","Data":"f31cfc03f43514ea96f707a711f8e76e26b651fcdfc96c102c82d5c39a159920"}
Jul 27 17:08:46.490473 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:46.490432    2566 generic.go:356] "Generic (PLEG): container finished" podID="4024487b-1e9f-477b-8eda-a1acfb2694e6" containerID="a9d464771112e84c33813b6eb4570f091d61e313165f6064ab42b00ff440ffe2" exitCode=0
Jul 27 17:08:46.491086 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:46.490467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q" event={"ID":"4024487b-1e9f-477b-8eda-a1acfb2694e6","Type":"ContainerDied","Data":"a9d464771112e84c33813b6eb4570f091d61e313165f6064ab42b00ff440ffe2"}
Jul 27 17:08:47.498019 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:47.497984    2566 generic.go:356] "Generic (PLEG): container finished" podID="4024487b-1e9f-477b-8eda-a1acfb2694e6" containerID="642186da985dabf65c628edfbfd3e4dbc445b26c3ed4c324da56211c5d00c008" exitCode=0
Jul 27 17:08:47.498450 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:47.498072    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q" event={"ID":"4024487b-1e9f-477b-8eda-a1acfb2694e6","Type":"ContainerDied","Data":"642186da985dabf65c628edfbfd3e4dbc445b26c3ed4c324da56211c5d00c008"}
Jul 27 17:08:48.629475 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:48.629450    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q"
Jul 27 17:08:48.730248 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:48.730219    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4024487b-1e9f-477b-8eda-a1acfb2694e6-util\") pod \"4024487b-1e9f-477b-8eda-a1acfb2694e6\" (UID: \"4024487b-1e9f-477b-8eda-a1acfb2694e6\") "
Jul 27 17:08:48.730423 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:48.730271    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkttr\" (UniqueName: \"kubernetes.io/projected/4024487b-1e9f-477b-8eda-a1acfb2694e6-kube-api-access-rkttr\") pod \"4024487b-1e9f-477b-8eda-a1acfb2694e6\" (UID: \"4024487b-1e9f-477b-8eda-a1acfb2694e6\") "
Jul 27 17:08:48.730423 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:48.730320    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4024487b-1e9f-477b-8eda-a1acfb2694e6-bundle\") pod \"4024487b-1e9f-477b-8eda-a1acfb2694e6\" (UID: \"4024487b-1e9f-477b-8eda-a1acfb2694e6\") "
Jul 27 17:08:48.730731 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:48.730709    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4024487b-1e9f-477b-8eda-a1acfb2694e6-bundle" (OuterVolumeSpecName: "bundle") pod "4024487b-1e9f-477b-8eda-a1acfb2694e6" (UID: "4024487b-1e9f-477b-8eda-a1acfb2694e6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:08:48.732410 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:48.732390    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4024487b-1e9f-477b-8eda-a1acfb2694e6-kube-api-access-rkttr" (OuterVolumeSpecName: "kube-api-access-rkttr") pod "4024487b-1e9f-477b-8eda-a1acfb2694e6" (UID: "4024487b-1e9f-477b-8eda-a1acfb2694e6"). InnerVolumeSpecName "kube-api-access-rkttr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:08:48.734517 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:48.734494    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4024487b-1e9f-477b-8eda-a1acfb2694e6-util" (OuterVolumeSpecName: "util") pod "4024487b-1e9f-477b-8eda-a1acfb2694e6" (UID: "4024487b-1e9f-477b-8eda-a1acfb2694e6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:08:48.831623 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:48.831559    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4024487b-1e9f-477b-8eda-a1acfb2694e6-util\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:08:48.831623 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:48.831584    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkttr\" (UniqueName: \"kubernetes.io/projected/4024487b-1e9f-477b-8eda-a1acfb2694e6-kube-api-access-rkttr\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:08:48.831623 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:48.831596    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4024487b-1e9f-477b-8eda-a1acfb2694e6-bundle\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:08:49.194578 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.194544    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lqwps"]
Jul 27 17:08:49.194897 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.194885    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4024487b-1e9f-477b-8eda-a1acfb2694e6" containerName="util"
Jul 27 17:08:49.194939 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.194898    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4024487b-1e9f-477b-8eda-a1acfb2694e6" containerName="util"
Jul 27 17:08:49.194939 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.194921    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4024487b-1e9f-477b-8eda-a1acfb2694e6" containerName="pull"
Jul 27 17:08:49.194939 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.194926    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4024487b-1e9f-477b-8eda-a1acfb2694e6" containerName="pull"
Jul 27 17:08:49.194939 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.194933    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4024487b-1e9f-477b-8eda-a1acfb2694e6" containerName="extract"
Jul 27 17:08:49.194939 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.194939    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4024487b-1e9f-477b-8eda-a1acfb2694e6" containerName="extract"
Jul 27 17:08:49.195087 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.194994    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4024487b-1e9f-477b-8eda-a1acfb2694e6" containerName="extract"
Jul 27 17:08:49.199333 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.199314    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lqwps"
Jul 27 17:08:49.203307 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.203287    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 27 17:08:49.204098 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.204080    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 27 17:08:49.204148 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.204103    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-f5chg\""
Jul 27 17:08:49.209702 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.209680    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lqwps"]
Jul 27 17:08:49.335733 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.335700    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/663313da-8306-4f24-ad8a-69469f5c2072-bound-sa-token\") pod \"cert-manager-55b4d954d5-lqwps\" (UID: \"663313da-8306-4f24-ad8a-69469f5c2072\") " pod="cert-manager/cert-manager-55b4d954d5-lqwps"
Jul 27 17:08:49.335901 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.335751    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4prz\" (UniqueName: \"kubernetes.io/projected/663313da-8306-4f24-ad8a-69469f5c2072-kube-api-access-f4prz\") pod \"cert-manager-55b4d954d5-lqwps\" (UID: \"663313da-8306-4f24-ad8a-69469f5c2072\") " pod="cert-manager/cert-manager-55b4d954d5-lqwps"
Jul 27 17:08:49.437063 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.437030    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/663313da-8306-4f24-ad8a-69469f5c2072-bound-sa-token\") pod \"cert-manager-55b4d954d5-lqwps\" (UID: \"663313da-8306-4f24-ad8a-69469f5c2072\") " pod="cert-manager/cert-manager-55b4d954d5-lqwps"
Jul 27 17:08:49.437215 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.437084    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4prz\" (UniqueName: \"kubernetes.io/projected/663313da-8306-4f24-ad8a-69469f5c2072-kube-api-access-f4prz\") pod \"cert-manager-55b4d954d5-lqwps\" (UID: \"663313da-8306-4f24-ad8a-69469f5c2072\") " pod="cert-manager/cert-manager-55b4d954d5-lqwps"
Jul 27 17:08:49.445026 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.444968    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/663313da-8306-4f24-ad8a-69469f5c2072-bound-sa-token\") pod \"cert-manager-55b4d954d5-lqwps\" (UID: \"663313da-8306-4f24-ad8a-69469f5c2072\") " pod="cert-manager/cert-manager-55b4d954d5-lqwps"
Jul 27 17:08:49.445230 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.445211    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4prz\" (UniqueName: \"kubernetes.io/projected/663313da-8306-4f24-ad8a-69469f5c2072-kube-api-access-f4prz\") pod \"cert-manager-55b4d954d5-lqwps\" (UID: \"663313da-8306-4f24-ad8a-69469f5c2072\") " pod="cert-manager/cert-manager-55b4d954d5-lqwps"
Jul 27 17:08:49.508078 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.508056    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lqwps"
Jul 27 17:08:49.509279 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.509249    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q" event={"ID":"4024487b-1e9f-477b-8eda-a1acfb2694e6","Type":"ContainerDied","Data":"f31cfc03f43514ea96f707a711f8e76e26b651fcdfc96c102c82d5c39a159920"}
Jul 27 17:08:49.509279 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.509271    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgdk2q"
Jul 27 17:08:49.509399 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.509301    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f31cfc03f43514ea96f707a711f8e76e26b651fcdfc96c102c82d5c39a159920"
Jul 27 17:08:49.638411 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:49.638384    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lqwps"]
Jul 27 17:08:49.640901 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:08:49.640868    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod663313da_8306_4f24_ad8a_69469f5c2072.slice/crio-63312e19c9812be9a60b0a2e4fc07203d31800a19c947161bfec24eb8fafd3b0 WatchSource:0}: Error finding container 63312e19c9812be9a60b0a2e4fc07203d31800a19c947161bfec24eb8fafd3b0: Status 404 returned error can't find the container with id 63312e19c9812be9a60b0a2e4fc07203d31800a19c947161bfec24eb8fafd3b0
Jul 27 17:08:50.516604 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:50.516564    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lqwps" event={"ID":"663313da-8306-4f24-ad8a-69469f5c2072","Type":"ContainerStarted","Data":"63312e19c9812be9a60b0a2e4fc07203d31800a19c947161bfec24eb8fafd3b0"}
Jul 27 17:08:52.527643 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:52.527609    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lqwps" event={"ID":"663313da-8306-4f24-ad8a-69469f5c2072","Type":"ContainerStarted","Data":"434eaf5288cc42a9d1fd4c82a347ed1be8c43cd26a3e67a6194e5a3e6ce3f011"}
Jul 27 17:08:52.546000 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:08:52.545955    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-lqwps" podStartSLOduration=0.88447645 podStartE2EDuration="3.545943028s" totalImagesPullingTime="2.661466578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:08:52.544685282 +0000 UTC m=+476.619949762" watchObservedRunningTime="2026-07-27 17:08:52.545943028 +0000 UTC m=+476.621207515"
Jul 27 17:09:02.528785 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:02.528707    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68c8965d4b-mchhr" podUID="ee8ddad4-64b3-43a8-8692-84043ba4c045" containerName="console" containerID="cri-o://2b51abcb5f8f924d38c48c47e80cf302e8cd7510a48df6de71dcae52c49ed1b7" gracePeriod=15
Jul 27 17:09:02.546412 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:02.546387    2566 patch_prober.go:28] interesting pod/console-68c8965d4b-mchhr container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.132.0.22:8443/health\": dial tcp 10.132.0.22:8443: connect: connection refused" start-of-body=
Jul 27 17:09:02.546499 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:02.546435    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-68c8965d4b-mchhr" podUID="ee8ddad4-64b3-43a8-8692-84043ba4c045" containerName="console" probeResult="failure" output="Get \"https://10.132.0.22:8443/health\": dial tcp 10.132.0.22:8443: connect: connection refused"
Jul 27 17:09:02.777600 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:02.777573    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:09:02.953903 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:02.953872    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-trusted-ca-bundle\") pod \"ee8ddad4-64b3-43a8-8692-84043ba4c045\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") "
Jul 27 17:09:02.954056 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:02.953910    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-oauth-serving-cert\") pod \"ee8ddad4-64b3-43a8-8692-84043ba4c045\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") "
Jul 27 17:09:02.954056 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:02.953965    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee8ddad4-64b3-43a8-8692-84043ba4c045-console-serving-cert\") pod \"ee8ddad4-64b3-43a8-8692-84043ba4c045\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") "
Jul 27 17:09:02.954056 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:02.953982    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee8ddad4-64b3-43a8-8692-84043ba4c045-console-oauth-config\") pod \"ee8ddad4-64b3-43a8-8692-84043ba4c045\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") "
Jul 27 17:09:02.954056 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:02.953998    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-console-config\") pod \"ee8ddad4-64b3-43a8-8692-84043ba4c045\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") "
Jul 27 17:09:02.954056 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:02.954016    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbqk5\" (UniqueName: \"kubernetes.io/projected/ee8ddad4-64b3-43a8-8692-84043ba4c045-kube-api-access-lbqk5\") pod \"ee8ddad4-64b3-43a8-8692-84043ba4c045\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") "
Jul 27 17:09:02.954342 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:02.954071    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-service-ca\") pod \"ee8ddad4-64b3-43a8-8692-84043ba4c045\" (UID: \"ee8ddad4-64b3-43a8-8692-84043ba4c045\") "
Jul 27 17:09:02.954546 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:02.954515    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ee8ddad4-64b3-43a8-8692-84043ba4c045" (UID: "ee8ddad4-64b3-43a8-8692-84043ba4c045"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:09:02.954678 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:02.954634    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-console-config" (OuterVolumeSpecName: "console-config") pod "ee8ddad4-64b3-43a8-8692-84043ba4c045" (UID: "ee8ddad4-64b3-43a8-8692-84043ba4c045"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:09:02.954678 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:02.954660    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-service-ca" (OuterVolumeSpecName: "service-ca") pod "ee8ddad4-64b3-43a8-8692-84043ba4c045" (UID: "ee8ddad4-64b3-43a8-8692-84043ba4c045"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:09:02.954678 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:02.954669    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ee8ddad4-64b3-43a8-8692-84043ba4c045" (UID: "ee8ddad4-64b3-43a8-8692-84043ba4c045"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:09:02.956200 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:02.956156    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee8ddad4-64b3-43a8-8692-84043ba4c045-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ee8ddad4-64b3-43a8-8692-84043ba4c045" (UID: "ee8ddad4-64b3-43a8-8692-84043ba4c045"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:09:02.956290 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:02.956202    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee8ddad4-64b3-43a8-8692-84043ba4c045-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ee8ddad4-64b3-43a8-8692-84043ba4c045" (UID: "ee8ddad4-64b3-43a8-8692-84043ba4c045"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:09:02.956343 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:02.956284    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8ddad4-64b3-43a8-8692-84043ba4c045-kube-api-access-lbqk5" (OuterVolumeSpecName: "kube-api-access-lbqk5") pod "ee8ddad4-64b3-43a8-8692-84043ba4c045" (UID: "ee8ddad4-64b3-43a8-8692-84043ba4c045"). InnerVolumeSpecName "kube-api-access-lbqk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:09:03.055500 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:03.055461    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-service-ca\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:09:03.055500 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:03.055492    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-trusted-ca-bundle\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:09:03.055500 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:03.055502    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-oauth-serving-cert\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:09:03.055732 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:03.055512    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee8ddad4-64b3-43a8-8692-84043ba4c045-console-serving-cert\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:09:03.055732 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:03.055521    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee8ddad4-64b3-43a8-8692-84043ba4c045-console-oauth-config\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:09:03.055732 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:03.055529    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee8ddad4-64b3-43a8-8692-84043ba4c045-console-config\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:09:03.055732 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:03.055537    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbqk5\" (UniqueName: \"kubernetes.io/projected/ee8ddad4-64b3-43a8-8692-84043ba4c045-kube-api-access-lbqk5\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:09:03.582686 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:03.582648    2566 generic.go:356] "Generic (PLEG): container finished" podID="ee8ddad4-64b3-43a8-8692-84043ba4c045" containerID="2b51abcb5f8f924d38c48c47e80cf302e8cd7510a48df6de71dcae52c49ed1b7" exitCode=0
Jul 27 17:09:03.583120 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:03.582692    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c8965d4b-mchhr"
Jul 27 17:09:03.583120 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:03.582728    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c8965d4b-mchhr" event={"ID":"ee8ddad4-64b3-43a8-8692-84043ba4c045","Type":"ContainerDied","Data":"2b51abcb5f8f924d38c48c47e80cf302e8cd7510a48df6de71dcae52c49ed1b7"}
Jul 27 17:09:03.583120 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:03.582787    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c8965d4b-mchhr" event={"ID":"ee8ddad4-64b3-43a8-8692-84043ba4c045","Type":"ContainerDied","Data":"16ad1f30927c96299145e243925f98fd821cb4c330aa6e2de7e76dd548495084"}
Jul 27 17:09:03.583120 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:03.582814    2566 scope.go:117] "RemoveContainer" containerID="2b51abcb5f8f924d38c48c47e80cf302e8cd7510a48df6de71dcae52c49ed1b7"
Jul 27 17:09:03.593655 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:03.593560    2566 scope.go:117] "RemoveContainer" containerID="2b51abcb5f8f924d38c48c47e80cf302e8cd7510a48df6de71dcae52c49ed1b7"
Jul 27 17:09:03.594209 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:09:03.594184    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b51abcb5f8f924d38c48c47e80cf302e8cd7510a48df6de71dcae52c49ed1b7\": container with ID starting with 2b51abcb5f8f924d38c48c47e80cf302e8cd7510a48df6de71dcae52c49ed1b7 not found: ID does not exist" containerID="2b51abcb5f8f924d38c48c47e80cf302e8cd7510a48df6de71dcae52c49ed1b7"
Jul 27 17:09:03.594304 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:03.594217    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b51abcb5f8f924d38c48c47e80cf302e8cd7510a48df6de71dcae52c49ed1b7"} err="failed to get container status \"2b51abcb5f8f924d38c48c47e80cf302e8cd7510a48df6de71dcae52c49ed1b7\": rpc error: code = NotFound desc = could not find container \"2b51abcb5f8f924d38c48c47e80cf302e8cd7510a48df6de71dcae52c49ed1b7\": container with ID starting with 2b51abcb5f8f924d38c48c47e80cf302e8cd7510a48df6de71dcae52c49ed1b7 not found: ID does not exist"
Jul 27 17:09:03.610386 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:03.607267    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68c8965d4b-mchhr"]
Jul 27 17:09:03.613599 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:03.613566    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68c8965d4b-mchhr"]
Jul 27 17:09:04.402521 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:04.402486    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee8ddad4-64b3-43a8-8692-84043ba4c045" path="/var/lib/kubelet/pods/ee8ddad4-64b3-43a8-8692-84043ba4c045/volumes"
Jul 27 17:09:19.012244 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.012208    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"]
Jul 27 17:09:19.012737 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.012721    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee8ddad4-64b3-43a8-8692-84043ba4c045" containerName="console"
Jul 27 17:09:19.012794 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.012741    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee8ddad4-64b3-43a8-8692-84043ba4c045" containerName="console"
Jul 27 17:09:19.012862 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.012851    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee8ddad4-64b3-43a8-8692-84043ba4c045" containerName="console"
Jul 27 17:09:19.016942 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.016920    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.019997 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.019941    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 17:09:19.019997 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.019985    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-qw4df\""
Jul 27 17:09:19.020181 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.019998    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 17:09:19.020307 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.020288    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 27 17:09:19.027254 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.027234    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"]
Jul 27 17:09:19.086189 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.086140    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5854c1ae-85a1-4d4f-8d45-792d72188f40-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.086356 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.086214    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5854c1ae-85a1-4d4f-8d45-792d72188f40-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.086356 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.086265    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5854c1ae-85a1-4d4f-8d45-792d72188f40-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.086356 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.086292    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5854c1ae-85a1-4d4f-8d45-792d72188f40-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.086356 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.086309    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzd44\" (UniqueName: \"kubernetes.io/projected/5854c1ae-85a1-4d4f-8d45-792d72188f40-kube-api-access-mzd44\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.086356 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.086348    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5854c1ae-85a1-4d4f-8d45-792d72188f40-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.086543 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.086365    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5854c1ae-85a1-4d4f-8d45-792d72188f40-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.086543 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.086382    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5854c1ae-85a1-4d4f-8d45-792d72188f40-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.086543 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.086467    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5854c1ae-85a1-4d4f-8d45-792d72188f40-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.186833 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.186799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5854c1ae-85a1-4d4f-8d45-792d72188f40-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.186833 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.186831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzd44\" (UniqueName: \"kubernetes.io/projected/5854c1ae-85a1-4d4f-8d45-792d72188f40-kube-api-access-mzd44\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.187069 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.186855    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5854c1ae-85a1-4d4f-8d45-792d72188f40-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.187069 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.186874    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5854c1ae-85a1-4d4f-8d45-792d72188f40-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.187069 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.186899    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5854c1ae-85a1-4d4f-8d45-792d72188f40-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.187069 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.186950    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5854c1ae-85a1-4d4f-8d45-792d72188f40-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.187069 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.186988    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5854c1ae-85a1-4d4f-8d45-792d72188f40-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.187339 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.187071    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5854c1ae-85a1-4d4f-8d45-792d72188f40-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.187339 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.187128    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5854c1ae-85a1-4d4f-8d45-792d72188f40-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.187448 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.187336    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5854c1ae-85a1-4d4f-8d45-792d72188f40-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.187448 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.187394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5854c1ae-85a1-4d4f-8d45-792d72188f40-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.187665 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.187641    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5854c1ae-85a1-4d4f-8d45-792d72188f40-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.187740 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.187671    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5854c1ae-85a1-4d4f-8d45-792d72188f40-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.187740 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.187653    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5854c1ae-85a1-4d4f-8d45-792d72188f40-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.189201 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.189157    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5854c1ae-85a1-4d4f-8d45-792d72188f40-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.189554 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.189534    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5854c1ae-85a1-4d4f-8d45-792d72188f40-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.194904 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.194886    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzd44\" (UniqueName: \"kubernetes.io/projected/5854c1ae-85a1-4d4f-8d45-792d72188f40-kube-api-access-mzd44\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.195031 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.195012    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5854c1ae-85a1-4d4f-8d45-792d72188f40-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c\" (UID: \"5854c1ae-85a1-4d4f-8d45-792d72188f40\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.327819 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.327734    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:19.462796 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.462769    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"]
Jul 27 17:09:19.465205 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:09:19.465161    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5854c1ae_85a1_4d4f_8d45_792d72188f40.slice/crio-bfc2d3644a3e0fbcc2783e17d1204edd225b1a0d2eb3f26458508d95ed077437 WatchSource:0}: Error finding container bfc2d3644a3e0fbcc2783e17d1204edd225b1a0d2eb3f26458508d95ed077437: Status 404 returned error can't find the container with id bfc2d3644a3e0fbcc2783e17d1204edd225b1a0d2eb3f26458508d95ed077437
Jul 27 17:09:19.657097 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:19.657010    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c" event={"ID":"5854c1ae-85a1-4d4f-8d45-792d72188f40","Type":"ContainerStarted","Data":"bfc2d3644a3e0fbcc2783e17d1204edd225b1a0d2eb3f26458508d95ed077437"}
Jul 27 17:09:22.159352 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:22.159305    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 17:09:22.159647 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:22.159386    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 17:09:22.159647 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:22.159437    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 17:09:22.674837 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:22.674793    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c" event={"ID":"5854c1ae-85a1-4d4f-8d45-792d72188f40","Type":"ContainerStarted","Data":"7ad1501a252bd0dc22b83afde524c6a7832788a621871b380715a8a8576e490e"}
Jul 27 17:09:22.694427 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:22.694378    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c" podStartSLOduration=2.002667858 podStartE2EDuration="4.694360629s" totalImagesPullingTime="2.691692771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:09:22.692998222 +0000 UTC m=+506.768262776" watchObservedRunningTime="2026-07-27 17:09:22.694360629 +0000 UTC m=+506.769625116"
Jul 27 17:09:23.328946 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:23.328906    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:23.333707 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:23.333676    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:23.679765 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:23.679681    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:23.680735 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:23.680708    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:09:26.287457 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.287421    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4"]
Jul 27 17:09:26.291248 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.291227    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4"
Jul 27 17:09:26.294145 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.294119    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 27 17:09:26.295240 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.295205    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 27 17:09:26.295240 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.295226    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:09:26.295240 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.295206    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 27 17:09:26.295561 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.295439    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 27 17:09:26.295561 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.295531    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-f9nmc\""
Jul 27 17:09:26.303107 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.303083    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4"]
Jul 27 17:09:26.353372 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.353336    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2-metrics-cert\") pod \"lws-controller-manager-58bd97cf4-r6dk4\" (UID: \"d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2\") " pod="openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4"
Jul 27 17:09:26.353544 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.353381    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2-manager-config\") pod \"lws-controller-manager-58bd97cf4-r6dk4\" (UID: \"d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2\") " pod="openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4"
Jul 27 17:09:26.353544 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.353398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2-cert\") pod \"lws-controller-manager-58bd97cf4-r6dk4\" (UID: \"d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2\") " pod="openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4"
Jul 27 17:09:26.353544 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.353500    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66tc\" (UniqueName: \"kubernetes.io/projected/d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2-kube-api-access-f66tc\") pod \"lws-controller-manager-58bd97cf4-r6dk4\" (UID: \"d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2\") " pod="openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4"
Jul 27 17:09:26.454527 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.454489    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2-metrics-cert\") pod \"lws-controller-manager-58bd97cf4-r6dk4\" (UID: \"d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2\") " pod="openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4"
Jul 27 17:09:26.454740 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.454546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2-manager-config\") pod \"lws-controller-manager-58bd97cf4-r6dk4\" (UID: \"d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2\") " pod="openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4"
Jul 27 17:09:26.454740 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.454659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2-cert\") pod \"lws-controller-manager-58bd97cf4-r6dk4\" (UID: \"d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2\") " pod="openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4"
Jul 27 17:09:26.454740 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.454730    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66tc\" (UniqueName: \"kubernetes.io/projected/d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2-kube-api-access-f66tc\") pod \"lws-controller-manager-58bd97cf4-r6dk4\" (UID: \"d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2\") " pod="openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4"
Jul 27 17:09:26.455317 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.455293    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2-manager-config\") pod \"lws-controller-manager-58bd97cf4-r6dk4\" (UID: \"d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2\") " pod="openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4"
Jul 27 17:09:26.457039 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.457011    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2-cert\") pod \"lws-controller-manager-58bd97cf4-r6dk4\" (UID: \"d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2\") " pod="openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4"
Jul 27 17:09:26.457120 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.457066    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2-metrics-cert\") pod \"lws-controller-manager-58bd97cf4-r6dk4\" (UID: \"d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2\") " pod="openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4"
Jul 27 17:09:26.472212 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.472188    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66tc\" (UniqueName: \"kubernetes.io/projected/d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2-kube-api-access-f66tc\") pod \"lws-controller-manager-58bd97cf4-r6dk4\" (UID: \"d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2\") " pod="openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4"
Jul 27 17:09:26.603822 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.603722    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4"
Jul 27 17:09:26.737722 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:26.737500    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4"]
Jul 27 17:09:26.739935 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:09:26.739901    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d9b23f_2053_4ec9_a65d_bfa2109dc8b2.slice/crio-e9a7fcafdc750c94f4385ed15663b0e65f6f25f287cacdcf1adf1e85fa4d3764 WatchSource:0}: Error finding container e9a7fcafdc750c94f4385ed15663b0e65f6f25f287cacdcf1adf1e85fa4d3764: Status 404 returned error can't find the container with id e9a7fcafdc750c94f4385ed15663b0e65f6f25f287cacdcf1adf1e85fa4d3764
Jul 27 17:09:27.702874 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:27.702833    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4" event={"ID":"d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2","Type":"ContainerStarted","Data":"e9a7fcafdc750c94f4385ed15663b0e65f6f25f287cacdcf1adf1e85fa4d3764"}
Jul 27 17:09:27.920772 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:27.920733    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw"]
Jul 27 17:09:27.924590 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:27.924559    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw"
Jul 27 17:09:27.927040 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:27.927018    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 27 17:09:27.927161 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:27.927059    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 27 17:09:27.927972 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:27.927950    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fmp6k\""
Jul 27 17:09:27.932306 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:27.932280    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw"]
Jul 27 17:09:27.967304 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:27.967226    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/78e32145-7ff7-443e-9df0-20d1530e1064-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw\" (UID: \"78e32145-7ff7-443e-9df0-20d1530e1064\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw"
Jul 27 17:09:27.967304 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:27.967292    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/78e32145-7ff7-443e-9df0-20d1530e1064-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw\" (UID: \"78e32145-7ff7-443e-9df0-20d1530e1064\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw"
Jul 27 17:09:27.967510 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:27.967323    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrd6d\" (UniqueName: \"kubernetes.io/projected/78e32145-7ff7-443e-9df0-20d1530e1064-kube-api-access-lrd6d\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw\" (UID: \"78e32145-7ff7-443e-9df0-20d1530e1064\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw"
Jul 27 17:09:28.019916 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.019877    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv"]
Jul 27 17:09:28.023984 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.023958    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv"
Jul 27 17:09:28.032867 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.032803    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv"]
Jul 27 17:09:28.068769 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.068735    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d21173-c6ab-41ac-a84a-20c805e7e84e-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv\" (UID: \"13d21173-c6ab-41ac-a84a-20c805e7e84e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv"
Jul 27 17:09:28.068940 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.068807    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13d21173-c6ab-41ac-a84a-20c805e7e84e-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv\" (UID: \"13d21173-c6ab-41ac-a84a-20c805e7e84e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv"
Jul 27 17:09:28.068940 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.068841    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/78e32145-7ff7-443e-9df0-20d1530e1064-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw\" (UID: \"78e32145-7ff7-443e-9df0-20d1530e1064\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw"
Jul 27 17:09:28.068940 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.068886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl6l9\" (UniqueName: \"kubernetes.io/projected/13d21173-c6ab-41ac-a84a-20c805e7e84e-kube-api-access-sl6l9\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv\" (UID: \"13d21173-c6ab-41ac-a84a-20c805e7e84e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv"
Jul 27 17:09:28.068940 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.068911    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/78e32145-7ff7-443e-9df0-20d1530e1064-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw\" (UID: \"78e32145-7ff7-443e-9df0-20d1530e1064\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw"
Jul 27 17:09:28.068940 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.068931    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrd6d\" (UniqueName: \"kubernetes.io/projected/78e32145-7ff7-443e-9df0-20d1530e1064-kube-api-access-lrd6d\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw\" (UID: \"78e32145-7ff7-443e-9df0-20d1530e1064\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw"
Jul 27 17:09:28.069282 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.069264    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/78e32145-7ff7-443e-9df0-20d1530e1064-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw\" (UID: \"78e32145-7ff7-443e-9df0-20d1530e1064\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw"
Jul 27 17:09:28.069322 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.069276    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/78e32145-7ff7-443e-9df0-20d1530e1064-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw\" (UID: \"78e32145-7ff7-443e-9df0-20d1530e1064\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw"
Jul 27 17:09:28.077729 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.077696    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrd6d\" (UniqueName: \"kubernetes.io/projected/78e32145-7ff7-443e-9df0-20d1530e1064-kube-api-access-lrd6d\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw\" (UID: \"78e32145-7ff7-443e-9df0-20d1530e1064\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw"
Jul 27 17:09:28.121320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.121282    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx"]
Jul 27 17:09:28.125243 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.125218    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx"
Jul 27 17:09:28.135117 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.135089    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx"]
Jul 27 17:09:28.170433 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.170394    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl6l9\" (UniqueName: \"kubernetes.io/projected/13d21173-c6ab-41ac-a84a-20c805e7e84e-kube-api-access-sl6l9\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv\" (UID: \"13d21173-c6ab-41ac-a84a-20c805e7e84e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv"
Jul 27 17:09:28.170604 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.170457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed6f10a7-2ca9-43cd-bb5a-215526f5b430-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx\" (UID: \"ed6f10a7-2ca9-43cd-bb5a-215526f5b430\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx"
Jul 27 17:09:28.170604 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.170495    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d21173-c6ab-41ac-a84a-20c805e7e84e-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv\" (UID: \"13d21173-c6ab-41ac-a84a-20c805e7e84e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv"
Jul 27 17:09:28.170604 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.170536    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed6f10a7-2ca9-43cd-bb5a-215526f5b430-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx\" (UID: \"ed6f10a7-2ca9-43cd-bb5a-215526f5b430\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx"
Jul 27 17:09:28.170604 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.170585    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r46n7\" (UniqueName: \"kubernetes.io/projected/ed6f10a7-2ca9-43cd-bb5a-215526f5b430-kube-api-access-r46n7\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx\" (UID: \"ed6f10a7-2ca9-43cd-bb5a-215526f5b430\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx"
Jul 27 17:09:28.170819 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.170616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13d21173-c6ab-41ac-a84a-20c805e7e84e-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv\" (UID: \"13d21173-c6ab-41ac-a84a-20c805e7e84e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv"
Jul 27 17:09:28.170990 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.170956    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d21173-c6ab-41ac-a84a-20c805e7e84e-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv\" (UID: \"13d21173-c6ab-41ac-a84a-20c805e7e84e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv"
Jul 27 17:09:28.171059 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.171000    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13d21173-c6ab-41ac-a84a-20c805e7e84e-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv\" (UID: \"13d21173-c6ab-41ac-a84a-20c805e7e84e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv"
Jul 27 17:09:28.179692 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.179661    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl6l9\" (UniqueName: \"kubernetes.io/projected/13d21173-c6ab-41ac-a84a-20c805e7e84e-kube-api-access-sl6l9\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv\" (UID: \"13d21173-c6ab-41ac-a84a-20c805e7e84e\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv"
Jul 27 17:09:28.219547 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.219469    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp"]
Jul 27 17:09:28.223934 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.223911    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp"
Jul 27 17:09:28.231437 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.231404    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp"]
Jul 27 17:09:28.237843 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.237816    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw"
Jul 27 17:09:28.271494 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.271457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qfnr\" (UniqueName: \"kubernetes.io/projected/9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331-kube-api-access-6qfnr\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp\" (UID: \"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp"
Jul 27 17:09:28.271709 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.271546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed6f10a7-2ca9-43cd-bb5a-215526f5b430-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx\" (UID: \"ed6f10a7-2ca9-43cd-bb5a-215526f5b430\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx"
Jul 27 17:09:28.271709 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.271565    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp\" (UID: \"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp"
Jul 27 17:09:28.271709 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.271609    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed6f10a7-2ca9-43cd-bb5a-215526f5b430-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx\" (UID: \"ed6f10a7-2ca9-43cd-bb5a-215526f5b430\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx"
Jul 27 17:09:28.271709 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.271678    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r46n7\" (UniqueName: \"kubernetes.io/projected/ed6f10a7-2ca9-43cd-bb5a-215526f5b430-kube-api-access-r46n7\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx\" (UID: \"ed6f10a7-2ca9-43cd-bb5a-215526f5b430\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx"
Jul 27 17:09:28.271937 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.271756    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp\" (UID: \"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp"
Jul 27 17:09:28.272096 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.272061    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed6f10a7-2ca9-43cd-bb5a-215526f5b430-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx\" (UID: \"ed6f10a7-2ca9-43cd-bb5a-215526f5b430\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx"
Jul 27 17:09:28.272096 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.272085    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed6f10a7-2ca9-43cd-bb5a-215526f5b430-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx\" (UID: \"ed6f10a7-2ca9-43cd-bb5a-215526f5b430\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx"
Jul 27 17:09:28.280717 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.280688    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r46n7\" (UniqueName: \"kubernetes.io/projected/ed6f10a7-2ca9-43cd-bb5a-215526f5b430-kube-api-access-r46n7\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx\" (UID: \"ed6f10a7-2ca9-43cd-bb5a-215526f5b430\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx"
Jul 27 17:09:28.337215 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.337163    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv"
Jul 27 17:09:28.373274 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.373230    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp\" (UID: \"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp"
Jul 27 17:09:28.373454 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.373282    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qfnr\" (UniqueName: \"kubernetes.io/projected/9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331-kube-api-access-6qfnr\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp\" (UID: \"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp"
Jul 27 17:09:28.373454 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.373348    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp\" (UID: \"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp"
Jul 27 17:09:28.373688 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.373664    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp\" (UID: \"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp"
Jul 27 17:09:28.373688 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.373681    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp\" (UID: \"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp"
Jul 27 17:09:28.382497 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.382468    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qfnr\" (UniqueName: \"kubernetes.io/projected/9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331-kube-api-access-6qfnr\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp\" (UID: \"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp"
Jul 27 17:09:28.438182 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.438128    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx"
Jul 27 17:09:28.537287 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.536400    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp"
Jul 27 17:09:28.590138 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.590073    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw"]
Jul 27 17:09:28.593689 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:09:28.593660    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e32145_7ff7_443e_9df0_20d1530e1064.slice/crio-bc8c469cfdc72356f5cc24388e394205aeed39e3791aac49504cc68255079695 WatchSource:0}: Error finding container bc8c469cfdc72356f5cc24388e394205aeed39e3791aac49504cc68255079695: Status 404 returned error can't find the container with id bc8c469cfdc72356f5cc24388e394205aeed39e3791aac49504cc68255079695
Jul 27 17:09:28.608947 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.608685    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv"]
Jul 27 17:09:28.666209 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.666161    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx"]
Jul 27 17:09:28.668760 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:09:28.668727    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6f10a7_2ca9_43cd_bb5a_215526f5b430.slice/crio-420b6ed04b123a041186b07f2f8d9c9f1b8b349c18288503a119b23cdfe3f6c5 WatchSource:0}: Error finding container 420b6ed04b123a041186b07f2f8d9c9f1b8b349c18288503a119b23cdfe3f6c5: Status 404 returned error can't find the container with id 420b6ed04b123a041186b07f2f8d9c9f1b8b349c18288503a119b23cdfe3f6c5
Jul 27 17:09:28.710559 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.710527    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4" event={"ID":"d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2","Type":"ContainerStarted","Data":"d10cdf65122949f02371d4388b1b2dfa93dba3063165643a52826d39f9f225c8"}
Jul 27 17:09:28.711007 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.710662    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4"
Jul 27 17:09:28.713837 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.713336    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv" event={"ID":"13d21173-c6ab-41ac-a84a-20c805e7e84e","Type":"ContainerStarted","Data":"c7d09ff8f39c829b60804c8ddb9071d0b074328aa6d0278bd84f4bdfa02e9354"}
Jul 27 17:09:28.713837 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.713386    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv" event={"ID":"13d21173-c6ab-41ac-a84a-20c805e7e84e","Type":"ContainerStarted","Data":"fcb7e56dd78bcf65b8ee24c752dc0912925510a2b74aa95c606619609970fac0"}
Jul 27 17:09:28.715750 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.715616    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp"]
Jul 27 17:09:28.715750 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.715668    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw" event={"ID":"78e32145-7ff7-443e-9df0-20d1530e1064","Type":"ContainerStarted","Data":"e0b9e86d98b9f5e1a7bdfab429b406c52416d136795b8fe0a559cbf2d2a82a4e"}
Jul 27 17:09:28.715750 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.715695    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw" event={"ID":"78e32145-7ff7-443e-9df0-20d1530e1064","Type":"ContainerStarted","Data":"bc8c469cfdc72356f5cc24388e394205aeed39e3791aac49504cc68255079695"}
Jul 27 17:09:28.717850 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.717825    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx" event={"ID":"ed6f10a7-2ca9-43cd-bb5a-215526f5b430","Type":"ContainerStarted","Data":"420b6ed04b123a041186b07f2f8d9c9f1b8b349c18288503a119b23cdfe3f6c5"}
Jul 27 17:09:28.729423 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:28.729342    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4" podStartSLOduration=0.979147047 podStartE2EDuration="2.729325104s" totalImagesPullingTime="1.750178057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:09:28.728646881 +0000 UTC m=+512.803911394" watchObservedRunningTime="2026-07-27 17:09:28.729325104 +0000 UTC m=+512.804589643"
Jul 27 17:09:28.731726 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:09:28.731702    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a62d39f_1a5f_4dbc_b9dd_b1e0c1f31331.slice/crio-636de414e9c9a89defed2fb0fc932db61d516729981e373e9acf83601b04d760 WatchSource:0}: Error finding container 636de414e9c9a89defed2fb0fc932db61d516729981e373e9acf83601b04d760: Status 404 returned error can't find the container with id 636de414e9c9a89defed2fb0fc932db61d516729981e373e9acf83601b04d760
Jul 27 17:09:29.725293 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:29.725258    2566 generic.go:356] "Generic (PLEG): container finished" podID="13d21173-c6ab-41ac-a84a-20c805e7e84e" containerID="c7d09ff8f39c829b60804c8ddb9071d0b074328aa6d0278bd84f4bdfa02e9354" exitCode=0
Jul 27 17:09:29.725661 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:29.725347    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv" event={"ID":"13d21173-c6ab-41ac-a84a-20c805e7e84e","Type":"ContainerDied","Data":"c7d09ff8f39c829b60804c8ddb9071d0b074328aa6d0278bd84f4bdfa02e9354"}
Jul 27 17:09:29.726922 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:29.726825    2566 generic.go:356] "Generic (PLEG): container finished" podID="78e32145-7ff7-443e-9df0-20d1530e1064" containerID="e0b9e86d98b9f5e1a7bdfab429b406c52416d136795b8fe0a559cbf2d2a82a4e" exitCode=0
Jul 27 17:09:29.726922 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:29.726897    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw" event={"ID":"78e32145-7ff7-443e-9df0-20d1530e1064","Type":"ContainerDied","Data":"e0b9e86d98b9f5e1a7bdfab429b406c52416d136795b8fe0a559cbf2d2a82a4e"}
Jul 27 17:09:29.728439 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:29.728413    2566 generic.go:356] "Generic (PLEG): container finished" podID="9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331" containerID="94480f13ab2c3d52202e4203f898cf91ca0fb8e2490c4f56c497a26d2aadf93b" exitCode=0
Jul 27 17:09:29.728562 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:29.728440    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp" event={"ID":"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331","Type":"ContainerDied","Data":"94480f13ab2c3d52202e4203f898cf91ca0fb8e2490c4f56c497a26d2aadf93b"}
Jul 27 17:09:29.728562 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:29.728488    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp" event={"ID":"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331","Type":"ContainerStarted","Data":"636de414e9c9a89defed2fb0fc932db61d516729981e373e9acf83601b04d760"}
Jul 27 17:09:29.730342 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:29.730323    2566 generic.go:356] "Generic (PLEG): container finished" podID="ed6f10a7-2ca9-43cd-bb5a-215526f5b430" containerID="737d637231fce41292996af6a2a240d8630d9fc12df2d9aa8ecdc52139c79b0e" exitCode=0
Jul 27 17:09:29.730437 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:29.730400    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx" event={"ID":"ed6f10a7-2ca9-43cd-bb5a-215526f5b430","Type":"ContainerDied","Data":"737d637231fce41292996af6a2a240d8630d9fc12df2d9aa8ecdc52139c79b0e"}
Jul 27 17:09:30.736922 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:30.736895    2566 generic.go:356] "Generic (PLEG): container finished" podID="13d21173-c6ab-41ac-a84a-20c805e7e84e" containerID="23001a046c6d8bed67e14a58063b9f90e1211c1846917630aefb968010748ee4" exitCode=0
Jul 27 17:09:30.737383 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:30.736972    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv" event={"ID":"13d21173-c6ab-41ac-a84a-20c805e7e84e","Type":"ContainerDied","Data":"23001a046c6d8bed67e14a58063b9f90e1211c1846917630aefb968010748ee4"}
Jul 27 17:09:30.738888 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:30.738866    2566 generic.go:356] "Generic (PLEG): container finished" podID="78e32145-7ff7-443e-9df0-20d1530e1064" containerID="bc0de4e75f99867161340d43640b11f9ce6309b8d831476b1e01730bc20abdcf" exitCode=0
Jul 27 17:09:30.738970 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:30.738946    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw" event={"ID":"78e32145-7ff7-443e-9df0-20d1530e1064","Type":"ContainerDied","Data":"bc0de4e75f99867161340d43640b11f9ce6309b8d831476b1e01730bc20abdcf"}
Jul 27 17:09:31.746002 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:31.745969    2566 generic.go:356] "Generic (PLEG): container finished" podID="13d21173-c6ab-41ac-a84a-20c805e7e84e" containerID="78b421e67865d568f89b8b705381a88e95028c40d9a1369bf90e14c43962063d" exitCode=0
Jul 27 17:09:31.746393 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:31.746044    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv" event={"ID":"13d21173-c6ab-41ac-a84a-20c805e7e84e","Type":"ContainerDied","Data":"78b421e67865d568f89b8b705381a88e95028c40d9a1369bf90e14c43962063d"}
Jul 27 17:09:31.748036 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:31.748009    2566 generic.go:356] "Generic (PLEG): container finished" podID="78e32145-7ff7-443e-9df0-20d1530e1064" containerID="fae45b914a100e070962d541f3cbb0e99d4315735dfba3b5f275189b4302d664" exitCode=0
Jul 27 17:09:31.748152 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:31.748089    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw" event={"ID":"78e32145-7ff7-443e-9df0-20d1530e1064","Type":"ContainerDied","Data":"fae45b914a100e070962d541f3cbb0e99d4315735dfba3b5f275189b4302d664"}
Jul 27 17:09:31.749732 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:31.749710    2566 generic.go:356] "Generic (PLEG): container finished" podID="9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331" containerID="ed145ad23cc25753627e0c1f3bb6ddb6814b9a984a6d85484524011ff0f8d7e7" exitCode=0
Jul 27 17:09:31.749857 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:31.749737    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp" event={"ID":"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331","Type":"ContainerDied","Data":"ed145ad23cc25753627e0c1f3bb6ddb6814b9a984a6d85484524011ff0f8d7e7"}
Jul 27 17:09:31.751679 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:31.751659    2566 generic.go:356] "Generic (PLEG): container finished" podID="ed6f10a7-2ca9-43cd-bb5a-215526f5b430" containerID="0a3117dd6ee6af56e94b0d513668e69b3d6793eb46d62784e012fbca3990249a" exitCode=0
Jul 27 17:09:31.751766 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:31.751703    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx" event={"ID":"ed6f10a7-2ca9-43cd-bb5a-215526f5b430","Type":"ContainerDied","Data":"0a3117dd6ee6af56e94b0d513668e69b3d6793eb46d62784e012fbca3990249a"}
Jul 27 17:09:32.759350 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:32.759310    2566 generic.go:356] "Generic (PLEG): container finished" podID="9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331" containerID="de63d2e260b9b1c3ee3dd888cad5c0b48ee85bcd1963566e5147767f52909991" exitCode=0
Jul 27 17:09:32.759739 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:32.759404    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp" event={"ID":"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331","Type":"ContainerDied","Data":"de63d2e260b9b1c3ee3dd888cad5c0b48ee85bcd1963566e5147767f52909991"}
Jul 27 17:09:32.761370 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:32.761347    2566 generic.go:356] "Generic (PLEG): container finished" podID="ed6f10a7-2ca9-43cd-bb5a-215526f5b430" containerID="3b2e01d037da3a2a1de2fc717236d7dc296e41ae281a96e9207e914978338df0" exitCode=0
Jul 27 17:09:32.761483 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:32.761431    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx" event={"ID":"ed6f10a7-2ca9-43cd-bb5a-215526f5b430","Type":"ContainerDied","Data":"3b2e01d037da3a2a1de2fc717236d7dc296e41ae281a96e9207e914978338df0"}
Jul 27 17:09:32.926409 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:32.926383    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw"
Jul 27 17:09:32.950948 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:32.950918    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv"
Jul 27 17:09:33.012770 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.012680    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrd6d\" (UniqueName: \"kubernetes.io/projected/78e32145-7ff7-443e-9df0-20d1530e1064-kube-api-access-lrd6d\") pod \"78e32145-7ff7-443e-9df0-20d1530e1064\" (UID: \"78e32145-7ff7-443e-9df0-20d1530e1064\") "
Jul 27 17:09:33.012929 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.012769    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl6l9\" (UniqueName: \"kubernetes.io/projected/13d21173-c6ab-41ac-a84a-20c805e7e84e-kube-api-access-sl6l9\") pod \"13d21173-c6ab-41ac-a84a-20c805e7e84e\" (UID: \"13d21173-c6ab-41ac-a84a-20c805e7e84e\") "
Jul 27 17:09:33.012929 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.012861    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/78e32145-7ff7-443e-9df0-20d1530e1064-util\") pod \"78e32145-7ff7-443e-9df0-20d1530e1064\" (UID: \"78e32145-7ff7-443e-9df0-20d1530e1064\") "
Jul 27 17:09:33.012929 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.012888    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/78e32145-7ff7-443e-9df0-20d1530e1064-bundle\") pod \"78e32145-7ff7-443e-9df0-20d1530e1064\" (UID: \"78e32145-7ff7-443e-9df0-20d1530e1064\") "
Jul 27 17:09:33.012929 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.012908    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13d21173-c6ab-41ac-a84a-20c805e7e84e-util\") pod \"13d21173-c6ab-41ac-a84a-20c805e7e84e\" (UID: \"13d21173-c6ab-41ac-a84a-20c805e7e84e\") "
Jul 27 17:09:33.013117 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.012938    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d21173-c6ab-41ac-a84a-20c805e7e84e-bundle\") pod \"13d21173-c6ab-41ac-a84a-20c805e7e84e\" (UID: \"13d21173-c6ab-41ac-a84a-20c805e7e84e\") "
Jul 27 17:09:33.013505 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.013472    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e32145-7ff7-443e-9df0-20d1530e1064-bundle" (OuterVolumeSpecName: "bundle") pod "78e32145-7ff7-443e-9df0-20d1530e1064" (UID: "78e32145-7ff7-443e-9df0-20d1530e1064"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:09:33.013608 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.013497    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d21173-c6ab-41ac-a84a-20c805e7e84e-bundle" (OuterVolumeSpecName: "bundle") pod "13d21173-c6ab-41ac-a84a-20c805e7e84e" (UID: "13d21173-c6ab-41ac-a84a-20c805e7e84e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:09:33.014878 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.014855    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d21173-c6ab-41ac-a84a-20c805e7e84e-kube-api-access-sl6l9" (OuterVolumeSpecName: "kube-api-access-sl6l9") pod "13d21173-c6ab-41ac-a84a-20c805e7e84e" (UID: "13d21173-c6ab-41ac-a84a-20c805e7e84e"). InnerVolumeSpecName "kube-api-access-sl6l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:09:33.015157 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.015139    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e32145-7ff7-443e-9df0-20d1530e1064-kube-api-access-lrd6d" (OuterVolumeSpecName: "kube-api-access-lrd6d") pod "78e32145-7ff7-443e-9df0-20d1530e1064" (UID: "78e32145-7ff7-443e-9df0-20d1530e1064"). InnerVolumeSpecName "kube-api-access-lrd6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:09:33.018607 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.018580    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e32145-7ff7-443e-9df0-20d1530e1064-util" (OuterVolumeSpecName: "util") pod "78e32145-7ff7-443e-9df0-20d1530e1064" (UID: "78e32145-7ff7-443e-9df0-20d1530e1064"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:09:33.019417 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.019392    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d21173-c6ab-41ac-a84a-20c805e7e84e-util" (OuterVolumeSpecName: "util") pod "13d21173-c6ab-41ac-a84a-20c805e7e84e" (UID: "13d21173-c6ab-41ac-a84a-20c805e7e84e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:09:33.114558 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.114521    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrd6d\" (UniqueName: \"kubernetes.io/projected/78e32145-7ff7-443e-9df0-20d1530e1064-kube-api-access-lrd6d\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:09:33.114558 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.114553    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl6l9\" (UniqueName: \"kubernetes.io/projected/13d21173-c6ab-41ac-a84a-20c805e7e84e-kube-api-access-sl6l9\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:09:33.114558 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.114563    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/78e32145-7ff7-443e-9df0-20d1530e1064-util\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:09:33.114790 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.114574    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/78e32145-7ff7-443e-9df0-20d1530e1064-bundle\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:09:33.114790 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.114582    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13d21173-c6ab-41ac-a84a-20c805e7e84e-util\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:09:33.114790 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.114590    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d21173-c6ab-41ac-a84a-20c805e7e84e-bundle\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:09:33.772236 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.772199    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv" event={"ID":"13d21173-c6ab-41ac-a84a-20c805e7e84e","Type":"ContainerDied","Data":"fcb7e56dd78bcf65b8ee24c752dc0912925510a2b74aa95c606619609970fac0"}
Jul 27 17:09:33.772236 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.772253    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcb7e56dd78bcf65b8ee24c752dc0912925510a2b74aa95c606619609970fac0"
Jul 27 17:09:33.772801 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.772215    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0zjbtv"
Jul 27 17:09:33.774185 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.774142    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw"
Jul 27 17:09:33.774296 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.774197    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbt4bnw" event={"ID":"78e32145-7ff7-443e-9df0-20d1530e1064","Type":"ContainerDied","Data":"bc8c469cfdc72356f5cc24388e394205aeed39e3791aac49504cc68255079695"}
Jul 27 17:09:33.774296 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.774239    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc8c469cfdc72356f5cc24388e394205aeed39e3791aac49504cc68255079695"
Jul 27 17:09:33.941749 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.941725    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx"
Jul 27 17:09:33.943923 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:33.943901    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp"
Jul 27 17:09:34.023362 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.023270    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331-bundle\") pod \"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331\" (UID: \"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331\") "
Jul 27 17:09:34.023362 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.023313    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331-util\") pod \"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331\" (UID: \"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331\") "
Jul 27 17:09:34.023586 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.023390    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r46n7\" (UniqueName: \"kubernetes.io/projected/ed6f10a7-2ca9-43cd-bb5a-215526f5b430-kube-api-access-r46n7\") pod \"ed6f10a7-2ca9-43cd-bb5a-215526f5b430\" (UID: \"ed6f10a7-2ca9-43cd-bb5a-215526f5b430\") "
Jul 27 17:09:34.023586 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.023429    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed6f10a7-2ca9-43cd-bb5a-215526f5b430-util\") pod \"ed6f10a7-2ca9-43cd-bb5a-215526f5b430\" (UID: \"ed6f10a7-2ca9-43cd-bb5a-215526f5b430\") "
Jul 27 17:09:34.023586 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.023474    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qfnr\" (UniqueName: \"kubernetes.io/projected/9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331-kube-api-access-6qfnr\") pod \"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331\" (UID: \"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331\") "
Jul 27 17:09:34.023586 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.023500    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed6f10a7-2ca9-43cd-bb5a-215526f5b430-bundle\") pod \"ed6f10a7-2ca9-43cd-bb5a-215526f5b430\" (UID: \"ed6f10a7-2ca9-43cd-bb5a-215526f5b430\") "
Jul 27 17:09:34.024435 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.024365    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331-bundle" (OuterVolumeSpecName: "bundle") pod "9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331" (UID: "9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:09:34.024631 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.024527    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed6f10a7-2ca9-43cd-bb5a-215526f5b430-bundle" (OuterVolumeSpecName: "bundle") pod "ed6f10a7-2ca9-43cd-bb5a-215526f5b430" (UID: "ed6f10a7-2ca9-43cd-bb5a-215526f5b430"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:09:34.027638 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.027607    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6f10a7-2ca9-43cd-bb5a-215526f5b430-kube-api-access-r46n7" (OuterVolumeSpecName: "kube-api-access-r46n7") pod "ed6f10a7-2ca9-43cd-bb5a-215526f5b430" (UID: "ed6f10a7-2ca9-43cd-bb5a-215526f5b430"). InnerVolumeSpecName "kube-api-access-r46n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:09:34.028717 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.028690    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331-kube-api-access-6qfnr" (OuterVolumeSpecName: "kube-api-access-6qfnr") pod "9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331" (UID: "9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331"). InnerVolumeSpecName "kube-api-access-6qfnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:09:34.032469 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.032442    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed6f10a7-2ca9-43cd-bb5a-215526f5b430-util" (OuterVolumeSpecName: "util") pod "ed6f10a7-2ca9-43cd-bb5a-215526f5b430" (UID: "ed6f10a7-2ca9-43cd-bb5a-215526f5b430"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:09:34.032594 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.032571    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331-util" (OuterVolumeSpecName: "util") pod "9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331" (UID: "9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:09:34.124824 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.124787    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r46n7\" (UniqueName: \"kubernetes.io/projected/ed6f10a7-2ca9-43cd-bb5a-215526f5b430-kube-api-access-r46n7\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:09:34.124824 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.124816    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed6f10a7-2ca9-43cd-bb5a-215526f5b430-util\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:09:34.124824 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.124826    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qfnr\" (UniqueName: \"kubernetes.io/projected/9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331-kube-api-access-6qfnr\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:09:34.124824 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.124835    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed6f10a7-2ca9-43cd-bb5a-215526f5b430-bundle\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:09:34.125089 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.124844    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331-bundle\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:09:34.125089 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.124851    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331-util\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:09:34.781441 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.781413    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp"
Jul 27 17:09:34.781441 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.781418    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053dxdqp" event={"ID":"9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331","Type":"ContainerDied","Data":"636de414e9c9a89defed2fb0fc932db61d516729981e373e9acf83601b04d760"}
Jul 27 17:09:34.781918 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.781467    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="636de414e9c9a89defed2fb0fc932db61d516729981e373e9acf83601b04d760"
Jul 27 17:09:34.783522 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.783493    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx" event={"ID":"ed6f10a7-2ca9-43cd-bb5a-215526f5b430","Type":"ContainerDied","Data":"420b6ed04b123a041186b07f2f8d9c9f1b8b349c18288503a119b23cdfe3f6c5"}
Jul 27 17:09:34.783522 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.783513    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183crrhgx"
Jul 27 17:09:34.783908 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:34.783541    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="420b6ed04b123a041186b07f2f8d9c9f1b8b349c18288503a119b23cdfe3f6c5"
Jul 27 17:09:39.733106 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:09:39.733071    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-58bd97cf4-r6dk4"
Jul 27 17:10:32.570959 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.570926    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-d5gnd"]
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571290    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed6f10a7-2ca9-43cd-bb5a-215526f5b430" containerName="util"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571302    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed6f10a7-2ca9-43cd-bb5a-215526f5b430" containerName="util"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571313    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d21173-c6ab-41ac-a84a-20c805e7e84e" containerName="pull"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571318    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d21173-c6ab-41ac-a84a-20c805e7e84e" containerName="pull"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571327    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e32145-7ff7-443e-9df0-20d1530e1064" containerName="extract"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571333    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e32145-7ff7-443e-9df0-20d1530e1064" containerName="extract"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571340    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed6f10a7-2ca9-43cd-bb5a-215526f5b430" containerName="extract"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571345    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed6f10a7-2ca9-43cd-bb5a-215526f5b430" containerName="extract"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571352    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d21173-c6ab-41ac-a84a-20c805e7e84e" containerName="extract"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571357    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d21173-c6ab-41ac-a84a-20c805e7e84e" containerName="extract"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571368    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e32145-7ff7-443e-9df0-20d1530e1064" containerName="util"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571373    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e32145-7ff7-443e-9df0-20d1530e1064" containerName="util"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571378    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331" containerName="util"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571383    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331" containerName="util"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571389    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331" containerName="pull"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571394    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331" containerName="pull"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571400    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331" containerName="extract"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571405    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331" containerName="extract"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571415    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed6f10a7-2ca9-43cd-bb5a-215526f5b430" containerName="pull"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571419    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed6f10a7-2ca9-43cd-bb5a-215526f5b430" containerName="pull"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571429    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d21173-c6ab-41ac-a84a-20c805e7e84e" containerName="util"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571435    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d21173-c6ab-41ac-a84a-20c805e7e84e" containerName="util"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571440    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e32145-7ff7-443e-9df0-20d1530e1064" containerName="pull"
Jul 27 17:10:32.571434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571445    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e32145-7ff7-443e-9df0-20d1530e1064" containerName="pull"
Jul 27 17:10:32.572250 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571496    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a62d39f-1a5f-4dbc-b9dd-b1e0c1f31331" containerName="extract"
Jul 27 17:10:32.572250 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571506    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="78e32145-7ff7-443e-9df0-20d1530e1064" containerName="extract"
Jul 27 17:10:32.572250 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571512    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed6f10a7-2ca9-43cd-bb5a-215526f5b430" containerName="extract"
Jul 27 17:10:32.572250 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.571519    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d21173-c6ab-41ac-a84a-20c805e7e84e" containerName="extract"
Jul 27 17:10:32.574330 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.574314    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-d5gnd"
Jul 27 17:10:32.576940 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.576922    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 17:10:32.577856 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.577823    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 17:10:32.577951 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.577823    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mqhsk\""
Jul 27 17:10:32.581365 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.581342    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-d5gnd"]
Jul 27 17:10:32.620607 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.620564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7pg9\" (UniqueName: \"kubernetes.io/projected/1bc654e8-6e22-4dff-846e-ae2842f217f2-kube-api-access-h7pg9\") pod \"authorino-b54df94b4-d5gnd\" (UID: \"1bc654e8-6e22-4dff-846e-ae2842f217f2\") " pod="kuadrant-system/authorino-b54df94b4-d5gnd"
Jul 27 17:10:32.689240 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.689204    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-mgdmn"]
Jul 27 17:10:32.692644 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.692627    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-mgdmn"
Jul 27 17:10:32.700107 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.699949    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-mgdmn"]
Jul 27 17:10:32.721902 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.721871    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7pg9\" (UniqueName: \"kubernetes.io/projected/1bc654e8-6e22-4dff-846e-ae2842f217f2-kube-api-access-h7pg9\") pod \"authorino-b54df94b4-d5gnd\" (UID: \"1bc654e8-6e22-4dff-846e-ae2842f217f2\") " pod="kuadrant-system/authorino-b54df94b4-d5gnd"
Jul 27 17:10:32.729434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.729406    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7pg9\" (UniqueName: \"kubernetes.io/projected/1bc654e8-6e22-4dff-846e-ae2842f217f2-kube-api-access-h7pg9\") pod \"authorino-b54df94b4-d5gnd\" (UID: \"1bc654e8-6e22-4dff-846e-ae2842f217f2\") " pod="kuadrant-system/authorino-b54df94b4-d5gnd"
Jul 27 17:10:32.823039 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.822944    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fkhw\" (UniqueName: \"kubernetes.io/projected/dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e-kube-api-access-2fkhw\") pod \"authorino-c548b5d7d-mgdmn\" (UID: \"dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e\") " pod="kuadrant-system/authorino-c548b5d7d-mgdmn"
Jul 27 17:10:32.884821 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.884777    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-d5gnd"
Jul 27 17:10:32.924514 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.924470    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkhw\" (UniqueName: \"kubernetes.io/projected/dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e-kube-api-access-2fkhw\") pod \"authorino-c548b5d7d-mgdmn\" (UID: \"dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e\") " pod="kuadrant-system/authorino-c548b5d7d-mgdmn"
Jul 27 17:10:32.932716 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:32.932656    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fkhw\" (UniqueName: \"kubernetes.io/projected/dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e-kube-api-access-2fkhw\") pod \"authorino-c548b5d7d-mgdmn\" (UID: \"dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e\") " pod="kuadrant-system/authorino-c548b5d7d-mgdmn"
Jul 27 17:10:33.002599 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:33.002568    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-mgdmn"
Jul 27 17:10:33.014881 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:33.014852    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-d5gnd"]
Jul 27 17:10:33.016248 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:10:33.016223    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bc654e8_6e22_4dff_846e_ae2842f217f2.slice/crio-db20eb9ea1d4466c1cdaa3819fe750f508254e2f56cec21704a71e130231231c WatchSource:0}: Error finding container db20eb9ea1d4466c1cdaa3819fe750f508254e2f56cec21704a71e130231231c: Status 404 returned error can't find the container with id db20eb9ea1d4466c1cdaa3819fe750f508254e2f56cec21704a71e130231231c
Jul 27 17:10:33.079777 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:33.079700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-d5gnd" event={"ID":"1bc654e8-6e22-4dff-846e-ae2842f217f2","Type":"ContainerStarted","Data":"db20eb9ea1d4466c1cdaa3819fe750f508254e2f56cec21704a71e130231231c"}
Jul 27 17:10:33.132288 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:33.132264    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-mgdmn"]
Jul 27 17:10:33.133909 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:10:33.133883    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddffb8fc0_9a7a_4c15_a616_dbb2ad0d3e7e.slice/crio-00d952bb0d948d8a17a400c5d6ada8f5328b117270e979ee30e78abe13bd200c WatchSource:0}: Error finding container 00d952bb0d948d8a17a400c5d6ada8f5328b117270e979ee30e78abe13bd200c: Status 404 returned error can't find the container with id 00d952bb0d948d8a17a400c5d6ada8f5328b117270e979ee30e78abe13bd200c
Jul 27 17:10:34.090807 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:34.090761    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-mgdmn" event={"ID":"dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e","Type":"ContainerStarted","Data":"00d952bb0d948d8a17a400c5d6ada8f5328b117270e979ee30e78abe13bd200c"}
Jul 27 17:10:37.109900 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:37.109842    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-d5gnd" event={"ID":"1bc654e8-6e22-4dff-846e-ae2842f217f2","Type":"ContainerStarted","Data":"32533e51044e862b16852e8120d3a3c19a34ce873c20f5684645b27d66b2c275"}
Jul 27 17:10:37.111459 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:37.111431    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-mgdmn" event={"ID":"dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e","Type":"ContainerStarted","Data":"271afce2f4977d4d6d7090459a37327aa2b0b67126f06c64918db9cbfe8cb354"}
Jul 27 17:10:37.125467 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:37.125415    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-d5gnd" podStartSLOduration=1.990165309 podStartE2EDuration="5.125399797s" totalImagesPullingTime="3.135234488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:10:37.123730169 +0000 UTC m=+581.198994656" watchObservedRunningTime="2026-07-27 17:10:37.125399797 +0000 UTC m=+581.200664284"
Jul 27 17:10:37.140487 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:37.140430    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-mgdmn" podStartSLOduration=2.111667846 podStartE2EDuration="5.140415085s" totalImagesPullingTime="3.028747239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:10:37.138374581 +0000 UTC m=+581.213639067" watchObservedRunningTime="2026-07-27 17:10:37.140415085 +0000 UTC m=+581.215679573"
Jul 27 17:10:37.166732 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:37.166695    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-d5gnd"]
Jul 27 17:10:39.121436 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:39.121368    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-d5gnd" podUID="1bc654e8-6e22-4dff-846e-ae2842f217f2" containerName="authorino" containerID="cri-o://32533e51044e862b16852e8120d3a3c19a34ce873c20f5684645b27d66b2c275" gracePeriod=30
Jul 27 17:10:39.376084 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:39.376025    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-d5gnd"
Jul 27 17:10:39.489753 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:39.489719    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7pg9\" (UniqueName: \"kubernetes.io/projected/1bc654e8-6e22-4dff-846e-ae2842f217f2-kube-api-access-h7pg9\") pod \"1bc654e8-6e22-4dff-846e-ae2842f217f2\" (UID: \"1bc654e8-6e22-4dff-846e-ae2842f217f2\") "
Jul 27 17:10:39.491874 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:39.491845    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc654e8-6e22-4dff-846e-ae2842f217f2-kube-api-access-h7pg9" (OuterVolumeSpecName: "kube-api-access-h7pg9") pod "1bc654e8-6e22-4dff-846e-ae2842f217f2" (UID: "1bc654e8-6e22-4dff-846e-ae2842f217f2"). InnerVolumeSpecName "kube-api-access-h7pg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:10:39.590881 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:39.590832    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7pg9\" (UniqueName: \"kubernetes.io/projected/1bc654e8-6e22-4dff-846e-ae2842f217f2-kube-api-access-h7pg9\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:10:40.128101 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:40.128064    2566 generic.go:356] "Generic (PLEG): container finished" podID="1bc654e8-6e22-4dff-846e-ae2842f217f2" containerID="32533e51044e862b16852e8120d3a3c19a34ce873c20f5684645b27d66b2c275" exitCode=0
Jul 27 17:10:40.128559 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:40.128106    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-d5gnd"
Jul 27 17:10:40.128559 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:40.128161    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-d5gnd" event={"ID":"1bc654e8-6e22-4dff-846e-ae2842f217f2","Type":"ContainerDied","Data":"32533e51044e862b16852e8120d3a3c19a34ce873c20f5684645b27d66b2c275"}
Jul 27 17:10:40.128559 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:40.128237    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-d5gnd" event={"ID":"1bc654e8-6e22-4dff-846e-ae2842f217f2","Type":"ContainerDied","Data":"db20eb9ea1d4466c1cdaa3819fe750f508254e2f56cec21704a71e130231231c"}
Jul 27 17:10:40.128559 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:40.128268    2566 scope.go:117] "RemoveContainer" containerID="32533e51044e862b16852e8120d3a3c19a34ce873c20f5684645b27d66b2c275"
Jul 27 17:10:40.137795 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:40.137764    2566 scope.go:117] "RemoveContainer" containerID="32533e51044e862b16852e8120d3a3c19a34ce873c20f5684645b27d66b2c275"
Jul 27 17:10:40.138059 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:10:40.138039    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32533e51044e862b16852e8120d3a3c19a34ce873c20f5684645b27d66b2c275\": container with ID starting with 32533e51044e862b16852e8120d3a3c19a34ce873c20f5684645b27d66b2c275 not found: ID does not exist" containerID="32533e51044e862b16852e8120d3a3c19a34ce873c20f5684645b27d66b2c275"
Jul 27 17:10:40.138101 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:40.138070    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32533e51044e862b16852e8120d3a3c19a34ce873c20f5684645b27d66b2c275"} err="failed to get container status \"32533e51044e862b16852e8120d3a3c19a34ce873c20f5684645b27d66b2c275\": rpc error: code = NotFound desc = could not find container \"32533e51044e862b16852e8120d3a3c19a34ce873c20f5684645b27d66b2c275\": container with ID starting with 32533e51044e862b16852e8120d3a3c19a34ce873c20f5684645b27d66b2c275 not found: ID does not exist"
Jul 27 17:10:40.149992 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:40.149964    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-d5gnd"]
Jul 27 17:10:40.153525 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:40.153502    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-d5gnd"]
Jul 27 17:10:40.403454 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:40.403376    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bc654e8-6e22-4dff-846e-ae2842f217f2" path="/var/lib/kubelet/pods/1bc654e8-6e22-4dff-846e-ae2842f217f2/volumes"
Jul 27 17:10:56.020507 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:56.020464    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cf4765b85-s6nzs"]
Jul 27 17:10:56.021023 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:56.021001    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bc654e8-6e22-4dff-846e-ae2842f217f2" containerName="authorino"
Jul 27 17:10:56.021105 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:56.021026    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bc654e8-6e22-4dff-846e-ae2842f217f2" containerName="authorino"
Jul 27 17:10:56.021193 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:56.021153    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bc654e8-6e22-4dff-846e-ae2842f217f2" containerName="authorino"
Jul 27 17:10:56.027626 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:56.027604    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-s6nzs"
Jul 27 17:10:56.030721 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:56.030695    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 27 17:10:56.031997 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:56.031974    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-s6nzs"]
Jul 27 17:10:56.143788 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:56.143753    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc-tls-cert\") pod \"authorino-7cf4765b85-s6nzs\" (UID: \"107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc\") " pod="kuadrant-system/authorino-7cf4765b85-s6nzs"
Jul 27 17:10:56.143989 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:56.143814    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87f8b\" (UniqueName: \"kubernetes.io/projected/107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc-kube-api-access-87f8b\") pod \"authorino-7cf4765b85-s6nzs\" (UID: \"107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc\") " pod="kuadrant-system/authorino-7cf4765b85-s6nzs"
Jul 27 17:10:56.244493 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:56.244460    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc-tls-cert\") pod \"authorino-7cf4765b85-s6nzs\" (UID: \"107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc\") " pod="kuadrant-system/authorino-7cf4765b85-s6nzs"
Jul 27 17:10:56.244681 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:56.244521    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87f8b\" (UniqueName: \"kubernetes.io/projected/107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc-kube-api-access-87f8b\") pod \"authorino-7cf4765b85-s6nzs\" (UID: \"107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc\") " pod="kuadrant-system/authorino-7cf4765b85-s6nzs"
Jul 27 17:10:56.246990 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:56.246968    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc-tls-cert\") pod \"authorino-7cf4765b85-s6nzs\" (UID: \"107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc\") " pod="kuadrant-system/authorino-7cf4765b85-s6nzs"
Jul 27 17:10:56.253539 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:56.253518    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87f8b\" (UniqueName: \"kubernetes.io/projected/107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc-kube-api-access-87f8b\") pod \"authorino-7cf4765b85-s6nzs\" (UID: \"107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc\") " pod="kuadrant-system/authorino-7cf4765b85-s6nzs"
Jul 27 17:10:56.337317 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:56.337239    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-s6nzs"
Jul 27 17:10:56.474406 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:56.474381    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-s6nzs"]
Jul 27 17:10:56.476555 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:10:56.476527    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod107c0c61_f4e0_4f44_b2e1_5ad3beaccfdc.slice/crio-f508707ff5aae3901f4ba31edc1b837368a21918245678d822e5731c1cf61c05 WatchSource:0}: Error finding container f508707ff5aae3901f4ba31edc1b837368a21918245678d822e5731c1cf61c05: Status 404 returned error can't find the container with id f508707ff5aae3901f4ba31edc1b837368a21918245678d822e5731c1cf61c05
Jul 27 17:10:57.216071 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:57.216040    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-s6nzs" event={"ID":"107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc","Type":"ContainerStarted","Data":"f508707ff5aae3901f4ba31edc1b837368a21918245678d822e5731c1cf61c05"}
Jul 27 17:10:57.303824 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:57.303791    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:10:57.304021 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:57.304003    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:10:58.223208 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:58.223150    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-s6nzs" event={"ID":"107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc","Type":"ContainerStarted","Data":"be2b62b8a6e2eda57333fb53c55c71bd66fe04b99198cb6e9d93ffc80a54fcc2"}
Jul 27 17:10:58.242077 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:58.242015    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cf4765b85-s6nzs" podStartSLOduration=1.51250942 podStartE2EDuration="2.241998123s" totalImagesPullingTime="729.488703ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:10:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:10:58.240743428 +0000 UTC m=+602.316007914" watchObservedRunningTime="2026-07-27 17:10:58.241998123 +0000 UTC m=+602.317262609"
Jul 27 17:10:58.268682 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:58.268638    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-mgdmn"]
Jul 27 17:10:58.268899 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:58.268872    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-mgdmn" podUID="dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e" containerName="authorino" containerID="cri-o://271afce2f4977d4d6d7090459a37327aa2b0b67126f06c64918db9cbfe8cb354" gracePeriod=30
Jul 27 17:10:58.521508 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:58.521482    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-mgdmn"
Jul 27 17:10:58.664946 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:58.664913    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fkhw\" (UniqueName: \"kubernetes.io/projected/dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e-kube-api-access-2fkhw\") pod \"dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e\" (UID: \"dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e\") "
Jul 27 17:10:58.667370 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:58.667342    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e-kube-api-access-2fkhw" (OuterVolumeSpecName: "kube-api-access-2fkhw") pod "dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e" (UID: "dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e"). InnerVolumeSpecName "kube-api-access-2fkhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:10:58.766092 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:58.766002    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fkhw\" (UniqueName: \"kubernetes.io/projected/dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e-kube-api-access-2fkhw\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:10:59.229024 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:59.228988    2566 generic.go:356] "Generic (PLEG): container finished" podID="dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e" containerID="271afce2f4977d4d6d7090459a37327aa2b0b67126f06c64918db9cbfe8cb354" exitCode=0
Jul 27 17:10:59.229479 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:59.229036    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-mgdmn"
Jul 27 17:10:59.229479 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:59.229078    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-mgdmn" event={"ID":"dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e","Type":"ContainerDied","Data":"271afce2f4977d4d6d7090459a37327aa2b0b67126f06c64918db9cbfe8cb354"}
Jul 27 17:10:59.229479 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:59.229125    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-mgdmn" event={"ID":"dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e","Type":"ContainerDied","Data":"00d952bb0d948d8a17a400c5d6ada8f5328b117270e979ee30e78abe13bd200c"}
Jul 27 17:10:59.229479 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:59.229144    2566 scope.go:117] "RemoveContainer" containerID="271afce2f4977d4d6d7090459a37327aa2b0b67126f06c64918db9cbfe8cb354"
Jul 27 17:10:59.238300 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:59.238280    2566 scope.go:117] "RemoveContainer" containerID="271afce2f4977d4d6d7090459a37327aa2b0b67126f06c64918db9cbfe8cb354"
Jul 27 17:10:59.238519 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:10:59.238503    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"271afce2f4977d4d6d7090459a37327aa2b0b67126f06c64918db9cbfe8cb354\": container with ID starting with 271afce2f4977d4d6d7090459a37327aa2b0b67126f06c64918db9cbfe8cb354 not found: ID does not exist" containerID="271afce2f4977d4d6d7090459a37327aa2b0b67126f06c64918db9cbfe8cb354"
Jul 27 17:10:59.238568 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:59.238527    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271afce2f4977d4d6d7090459a37327aa2b0b67126f06c64918db9cbfe8cb354"} err="failed to get container status \"271afce2f4977d4d6d7090459a37327aa2b0b67126f06c64918db9cbfe8cb354\": rpc error: code = NotFound desc = could not find container \"271afce2f4977d4d6d7090459a37327aa2b0b67126f06c64918db9cbfe8cb354\": container with ID starting with 271afce2f4977d4d6d7090459a37327aa2b0b67126f06c64918db9cbfe8cb354 not found: ID does not exist"
Jul 27 17:10:59.252731 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:59.252708    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-mgdmn"]
Jul 27 17:10:59.258503 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:10:59.258484    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-mgdmn"]
Jul 27 17:11:00.402945 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:00.402912    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e" path="/var/lib/kubelet/pods/dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e/volumes"
Jul 27 17:11:07.945861 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:07.945825    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx"]
Jul 27 17:11:07.946346 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:07.946042    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx" podUID="57f1804c-717f-46a4-8b1c-8ddfd1af84e6" containerName="manager" containerID="cri-o://af5b940e57fa39c1d54541ca7f340434cbd5c168eb1b5717c4c418c6b69dfb32" gracePeriod=10
Jul 27 17:11:07.970664 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:07.970639    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6ccfdc6bb8-tsbd7"]
Jul 27 17:11:07.971012 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:07.971000    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e" containerName="authorino"
Jul 27 17:11:07.971054 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:07.971013    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e" containerName="authorino"
Jul 27 17:11:07.971091 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:07.971072    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dffb8fc0-9a7a-4c15-a616-dbb2ad0d3e7e" containerName="authorino"
Jul 27 17:11:07.974559 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:07.974542    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6ccfdc6bb8-tsbd7"
Jul 27 17:11:07.982126 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:07.982096    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6ccfdc6bb8-tsbd7"]
Jul 27 17:11:08.039556 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.039521    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6edde0d5-156d-4368-91de-48b54e7b7087-cert\") pod \"kserve-controller-manager-6ccfdc6bb8-tsbd7\" (UID: \"6edde0d5-156d-4368-91de-48b54e7b7087\") " pod="kserve/kserve-controller-manager-6ccfdc6bb8-tsbd7"
Jul 27 17:11:08.039730 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.039581    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpq62\" (UniqueName: \"kubernetes.io/projected/6edde0d5-156d-4368-91de-48b54e7b7087-kube-api-access-mpq62\") pod \"kserve-controller-manager-6ccfdc6bb8-tsbd7\" (UID: \"6edde0d5-156d-4368-91de-48b54e7b7087\") " pod="kserve/kserve-controller-manager-6ccfdc6bb8-tsbd7"
Jul 27 17:11:08.140668 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.140637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpq62\" (UniqueName: \"kubernetes.io/projected/6edde0d5-156d-4368-91de-48b54e7b7087-kube-api-access-mpq62\") pod \"kserve-controller-manager-6ccfdc6bb8-tsbd7\" (UID: \"6edde0d5-156d-4368-91de-48b54e7b7087\") " pod="kserve/kserve-controller-manager-6ccfdc6bb8-tsbd7"
Jul 27 17:11:08.140850 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.140770    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6edde0d5-156d-4368-91de-48b54e7b7087-cert\") pod \"kserve-controller-manager-6ccfdc6bb8-tsbd7\" (UID: \"6edde0d5-156d-4368-91de-48b54e7b7087\") " pod="kserve/kserve-controller-manager-6ccfdc6bb8-tsbd7"
Jul 27 17:11:08.143526 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.143497    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6edde0d5-156d-4368-91de-48b54e7b7087-cert\") pod \"kserve-controller-manager-6ccfdc6bb8-tsbd7\" (UID: \"6edde0d5-156d-4368-91de-48b54e7b7087\") " pod="kserve/kserve-controller-manager-6ccfdc6bb8-tsbd7"
Jul 27 17:11:08.149199 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.149147    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpq62\" (UniqueName: \"kubernetes.io/projected/6edde0d5-156d-4368-91de-48b54e7b7087-kube-api-access-mpq62\") pod \"kserve-controller-manager-6ccfdc6bb8-tsbd7\" (UID: \"6edde0d5-156d-4368-91de-48b54e7b7087\") " pod="kserve/kserve-controller-manager-6ccfdc6bb8-tsbd7"
Jul 27 17:11:08.207715 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.207647    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx"
Jul 27 17:11:08.241980 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.241945    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq57m\" (UniqueName: \"kubernetes.io/projected/57f1804c-717f-46a4-8b1c-8ddfd1af84e6-kube-api-access-mq57m\") pod \"57f1804c-717f-46a4-8b1c-8ddfd1af84e6\" (UID: \"57f1804c-717f-46a4-8b1c-8ddfd1af84e6\") "
Jul 27 17:11:08.241980 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.241983    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57f1804c-717f-46a4-8b1c-8ddfd1af84e6-cert\") pod \"57f1804c-717f-46a4-8b1c-8ddfd1af84e6\" (UID: \"57f1804c-717f-46a4-8b1c-8ddfd1af84e6\") "
Jul 27 17:11:08.244295 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.244256    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f1804c-717f-46a4-8b1c-8ddfd1af84e6-kube-api-access-mq57m" (OuterVolumeSpecName: "kube-api-access-mq57m") pod "57f1804c-717f-46a4-8b1c-8ddfd1af84e6" (UID: "57f1804c-717f-46a4-8b1c-8ddfd1af84e6"). InnerVolumeSpecName "kube-api-access-mq57m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:11:08.244410 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.244329    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57f1804c-717f-46a4-8b1c-8ddfd1af84e6-cert" (OuterVolumeSpecName: "cert") pod "57f1804c-717f-46a4-8b1c-8ddfd1af84e6" (UID: "57f1804c-717f-46a4-8b1c-8ddfd1af84e6"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:11:08.283266 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.283228    2566 generic.go:356] "Generic (PLEG): container finished" podID="57f1804c-717f-46a4-8b1c-8ddfd1af84e6" containerID="af5b940e57fa39c1d54541ca7f340434cbd5c168eb1b5717c4c418c6b69dfb32" exitCode=0
Jul 27 17:11:08.283266 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.283278    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx"
Jul 27 17:11:08.283467 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.283282    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx" event={"ID":"57f1804c-717f-46a4-8b1c-8ddfd1af84e6","Type":"ContainerDied","Data":"af5b940e57fa39c1d54541ca7f340434cbd5c168eb1b5717c4c418c6b69dfb32"}
Jul 27 17:11:08.283467 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.283338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx" event={"ID":"57f1804c-717f-46a4-8b1c-8ddfd1af84e6","Type":"ContainerDied","Data":"19842464aa8c84acd35b3d1cdf04c7e07e99a1900a6076459c84b9966baf6363"}
Jul 27 17:11:08.283467 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.283367    2566 scope.go:117] "RemoveContainer" containerID="af5b940e57fa39c1d54541ca7f340434cbd5c168eb1b5717c4c418c6b69dfb32"
Jul 27 17:11:08.292444 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.292425    2566 scope.go:117] "RemoveContainer" containerID="af5b940e57fa39c1d54541ca7f340434cbd5c168eb1b5717c4c418c6b69dfb32"
Jul 27 17:11:08.292709 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:11:08.292689    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af5b940e57fa39c1d54541ca7f340434cbd5c168eb1b5717c4c418c6b69dfb32\": container with ID starting with af5b940e57fa39c1d54541ca7f340434cbd5c168eb1b5717c4c418c6b69dfb32 not found: ID does not exist" containerID="af5b940e57fa39c1d54541ca7f340434cbd5c168eb1b5717c4c418c6b69dfb32"
Jul 27 17:11:08.292757 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.292719    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af5b940e57fa39c1d54541ca7f340434cbd5c168eb1b5717c4c418c6b69dfb32"} err="failed to get container status \"af5b940e57fa39c1d54541ca7f340434cbd5c168eb1b5717c4c418c6b69dfb32\": rpc error: code = NotFound desc = could not find container \"af5b940e57fa39c1d54541ca7f340434cbd5c168eb1b5717c4c418c6b69dfb32\": container with ID starting with af5b940e57fa39c1d54541ca7f340434cbd5c168eb1b5717c4c418c6b69dfb32 not found: ID does not exist"
Jul 27 17:11:08.305137 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.305105    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx"]
Jul 27 17:11:08.309183 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.309145    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx"]
Jul 27 17:11:08.343625 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.343588    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq57m\" (UniqueName: \"kubernetes.io/projected/57f1804c-717f-46a4-8b1c-8ddfd1af84e6-kube-api-access-mq57m\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:11:08.343792 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.343619    2566 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57f1804c-717f-46a4-8b1c-8ddfd1af84e6-cert\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:11:08.343792 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.343711    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6ccfdc6bb8-tsbd7"
Jul 27 17:11:08.405633 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.405597    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57f1804c-717f-46a4-8b1c-8ddfd1af84e6" path="/var/lib/kubelet/pods/57f1804c-717f-46a4-8b1c-8ddfd1af84e6/volumes"
Jul 27 17:11:08.478071 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.478046    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6ccfdc6bb8-tsbd7"]
Jul 27 17:11:08.483006 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:11:08.482964    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6edde0d5_156d_4368_91de_48b54e7b7087.slice/crio-e2c61edd517f9eb6bc4c354350c7c0637d232f1f4f739dd2b0c705fbec38210e WatchSource:0}: Error finding container e2c61edd517f9eb6bc4c354350c7c0637d232f1f4f739dd2b0c705fbec38210e: Status 404 returned error can't find the container with id e2c61edd517f9eb6bc4c354350c7c0637d232f1f4f739dd2b0c705fbec38210e
Jul 27 17:11:08.483959 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:08.483941    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:11:09.290294 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:09.290189    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6ccfdc6bb8-tsbd7" event={"ID":"6edde0d5-156d-4368-91de-48b54e7b7087","Type":"ContainerStarted","Data":"f7a843824b6a8f0fe3d3816e9106af0083ae0804c49d3cf0a652f9babaf6d54c"}
Jul 27 17:11:09.290294 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:09.290243    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6ccfdc6bb8-tsbd7" event={"ID":"6edde0d5-156d-4368-91de-48b54e7b7087","Type":"ContainerStarted","Data":"e2c61edd517f9eb6bc4c354350c7c0637d232f1f4f739dd2b0c705fbec38210e"}
Jul 27 17:11:09.290294 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:09.290269    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6ccfdc6bb8-tsbd7"
Jul 27 17:11:13.183149 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:13.183093    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve/kserve-controller-manager-6ccfdc6bb8-mzzcx" podUID="57f1804c-717f-46a4-8b1c-8ddfd1af84e6" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 17:11:40.302310 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:40.302276    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6ccfdc6bb8-tsbd7"
Jul 27 17:11:40.320408 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:40.320355    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6ccfdc6bb8-tsbd7" podStartSLOduration=32.838140554 podStartE2EDuration="33.32034036s" totalImagesPullingTime="482.199806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:11:09.306123035 +0000 UTC m=+613.381387533" watchObservedRunningTime="2026-07-27 17:11:40.32034036 +0000 UTC m=+644.395604845"
Jul 27 17:11:59.602226 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:59.602192    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-9m24t"]
Jul 27 17:11:59.602593 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:59.602550    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57f1804c-717f-46a4-8b1c-8ddfd1af84e6" containerName="manager"
Jul 27 17:11:59.602593 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:59.602561    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="57f1804c-717f-46a4-8b1c-8ddfd1af84e6" containerName="manager"
Jul 27 17:11:59.602665 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:59.602635    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="57f1804c-717f-46a4-8b1c-8ddfd1af84e6" containerName="manager"
Jul 27 17:11:59.606978 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:59.606962    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9m24t"
Jul 27 17:11:59.609572 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:59.609543    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 27 17:11:59.610633 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:59.610607    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-vl2qf\""
Jul 27 17:11:59.612521 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:59.612497    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9m24t"]
Jul 27 17:11:59.697400 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:59.697364    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvldg\" (UniqueName: \"kubernetes.io/projected/e66aa8fe-8604-4b2e-bcb8-f171a15c374e-kube-api-access-nvldg\") pod \"s3-init-9m24t\" (UID: \"e66aa8fe-8604-4b2e-bcb8-f171a15c374e\") " pod="kserve/s3-init-9m24t"
Jul 27 17:11:59.697590 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:59.697423    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66aa8fe-8604-4b2e-bcb8-f171a15c374e-model-cache\") pod \"s3-init-9m24t\" (UID: \"e66aa8fe-8604-4b2e-bcb8-f171a15c374e\") " pod="kserve/s3-init-9m24t"
Jul 27 17:11:59.798823 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:59.798781    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvldg\" (UniqueName: \"kubernetes.io/projected/e66aa8fe-8604-4b2e-bcb8-f171a15c374e-kube-api-access-nvldg\") pod \"s3-init-9m24t\" (UID: \"e66aa8fe-8604-4b2e-bcb8-f171a15c374e\") " pod="kserve/s3-init-9m24t"
Jul 27 17:11:59.798995 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:59.798833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66aa8fe-8604-4b2e-bcb8-f171a15c374e-model-cache\") pod \"s3-init-9m24t\" (UID: \"e66aa8fe-8604-4b2e-bcb8-f171a15c374e\") " pod="kserve/s3-init-9m24t"
Jul 27 17:11:59.799230 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:59.799214    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66aa8fe-8604-4b2e-bcb8-f171a15c374e-model-cache\") pod \"s3-init-9m24t\" (UID: \"e66aa8fe-8604-4b2e-bcb8-f171a15c374e\") " pod="kserve/s3-init-9m24t"
Jul 27 17:11:59.807194 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:59.807149    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvldg\" (UniqueName: \"kubernetes.io/projected/e66aa8fe-8604-4b2e-bcb8-f171a15c374e-kube-api-access-nvldg\") pod \"s3-init-9m24t\" (UID: \"e66aa8fe-8604-4b2e-bcb8-f171a15c374e\") " pod="kserve/s3-init-9m24t"
Jul 27 17:11:59.917920 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:11:59.917811    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9m24t"
Jul 27 17:12:00.051814 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:12:00.051788    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9m24t"]
Jul 27 17:12:00.053789 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:12:00.053752    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode66aa8fe_8604_4b2e_bcb8_f171a15c374e.slice/crio-aad2527b1fee474bdd1006816547cebf3ea9e9b3129565ba3027a6209a845108 WatchSource:0}: Error finding container aad2527b1fee474bdd1006816547cebf3ea9e9b3129565ba3027a6209a845108: Status 404 returned error can't find the container with id aad2527b1fee474bdd1006816547cebf3ea9e9b3129565ba3027a6209a845108
Jul 27 17:12:00.551951 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:12:00.551911    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9m24t" event={"ID":"e66aa8fe-8604-4b2e-bcb8-f171a15c374e","Type":"ContainerStarted","Data":"aad2527b1fee474bdd1006816547cebf3ea9e9b3129565ba3027a6209a845108"}
Jul 27 17:12:04.575434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:12:04.575398    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9m24t" event={"ID":"e66aa8fe-8604-4b2e-bcb8-f171a15c374e","Type":"ContainerStarted","Data":"1a26d4de72532158bf576516cc4be320d896b096d95aca3581d5dc2b07e4d4db"}
Jul 27 17:12:07.592969 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:12:07.592934    2566 generic.go:356] "Generic (PLEG): container finished" podID="e66aa8fe-8604-4b2e-bcb8-f171a15c374e" containerID="1a26d4de72532158bf576516cc4be320d896b096d95aca3581d5dc2b07e4d4db" exitCode=0
Jul 27 17:12:07.593402 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:12:07.593010    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9m24t" event={"ID":"e66aa8fe-8604-4b2e-bcb8-f171a15c374e","Type":"ContainerDied","Data":"1a26d4de72532158bf576516cc4be320d896b096d95aca3581d5dc2b07e4d4db"}
Jul 27 17:12:12.623768 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:12:12.623729    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9m24t" event={"ID":"e66aa8fe-8604-4b2e-bcb8-f171a15c374e","Type":"ContainerStarted","Data":"ede3f99b3c4feef02ad89da1eed3317666c398fca3855d4621471dda72d53062"}
Jul 27 17:12:12.639707 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:12:12.639663    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-9m24t" podStartSLOduration=1.5522214399999998 podStartE2EDuration="13.639641117s" totalImagesPullingTime="8.847617855s" totalInitContainerRuntime="3.239801822s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:11:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:12:12.63950516 +0000 UTC m=+676.714769647" watchObservedRunningTime="2026-07-27 17:12:12.639641117 +0000 UTC m=+676.714905607"
Jul 27 17:13:17.964559 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:17.964523    2566 generic.go:356] "Generic (PLEG): container finished" podID="e66aa8fe-8604-4b2e-bcb8-f171a15c374e" containerID="ede3f99b3c4feef02ad89da1eed3317666c398fca3855d4621471dda72d53062" exitCode=0
Jul 27 17:13:17.965049 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:17.964598    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9m24t" event={"ID":"e66aa8fe-8604-4b2e-bcb8-f171a15c374e","Type":"ContainerDied","Data":"ede3f99b3c4feef02ad89da1eed3317666c398fca3855d4621471dda72d53062"}
Jul 27 17:13:19.105419 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:19.105394    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9m24t"
Jul 27 17:13:19.113893 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:19.113869    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvldg\" (UniqueName: \"kubernetes.io/projected/e66aa8fe-8604-4b2e-bcb8-f171a15c374e-kube-api-access-nvldg\") pod \"e66aa8fe-8604-4b2e-bcb8-f171a15c374e\" (UID: \"e66aa8fe-8604-4b2e-bcb8-f171a15c374e\") "
Jul 27 17:13:19.113957 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:19.113921    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66aa8fe-8604-4b2e-bcb8-f171a15c374e-model-cache\") pod \"e66aa8fe-8604-4b2e-bcb8-f171a15c374e\" (UID: \"e66aa8fe-8604-4b2e-bcb8-f171a15c374e\") "
Jul 27 17:13:19.115809 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:19.115780    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66aa8fe-8604-4b2e-bcb8-f171a15c374e-kube-api-access-nvldg" (OuterVolumeSpecName: "kube-api-access-nvldg") pod "e66aa8fe-8604-4b2e-bcb8-f171a15c374e" (UID: "e66aa8fe-8604-4b2e-bcb8-f171a15c374e"). InnerVolumeSpecName "kube-api-access-nvldg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:13:19.134287 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:19.134259    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66aa8fe-8604-4b2e-bcb8-f171a15c374e-model-cache" (OuterVolumeSpecName: "model-cache") pod "e66aa8fe-8604-4b2e-bcb8-f171a15c374e" (UID: "e66aa8fe-8604-4b2e-bcb8-f171a15c374e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:13:19.215248 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:19.215212    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvldg\" (UniqueName: \"kubernetes.io/projected/e66aa8fe-8604-4b2e-bcb8-f171a15c374e-kube-api-access-nvldg\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:13:19.215248 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:19.215243    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66aa8fe-8604-4b2e-bcb8-f171a15c374e-model-cache\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:13:19.977076 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:19.977042    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9m24t" event={"ID":"e66aa8fe-8604-4b2e-bcb8-f171a15c374e","Type":"ContainerDied","Data":"aad2527b1fee474bdd1006816547cebf3ea9e9b3129565ba3027a6209a845108"}
Jul 27 17:13:19.977311 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:19.977084    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aad2527b1fee474bdd1006816547cebf3ea9e9b3129565ba3027a6209a845108"
Jul 27 17:13:19.977311 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:19.977052    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9m24t"
Jul 27 17:13:53.585761 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.585725    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"]
Jul 27 17:13:53.586296 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.586279    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e66aa8fe-8604-4b2e-bcb8-f171a15c374e" containerName="download-hf-model"
Jul 27 17:13:53.586349 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.586300    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e66aa8fe-8604-4b2e-bcb8-f171a15c374e" containerName="download-hf-model"
Jul 27 17:13:53.586349 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.586319    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e66aa8fe-8604-4b2e-bcb8-f171a15c374e" containerName="s3-init"
Jul 27 17:13:53.586349 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.586327    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e66aa8fe-8604-4b2e-bcb8-f171a15c374e" containerName="s3-init"
Jul 27 17:13:53.586498 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.586487    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e66aa8fe-8604-4b2e-bcb8-f171a15c374e" containerName="s3-init"
Jul 27 17:13:53.589902 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.589884    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:53.592344 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.592324    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Jul 27 17:13:53.593533 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.593506    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-bxkk9\""
Jul 27 17:13:53.593625 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.593568    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Jul 27 17:13:53.593625 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.593568    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Jul 27 17:13:53.597376 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.597358    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"]
Jul 27 17:13:53.709587 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.709550    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-bqqvc\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:53.709746 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.709595    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c9c530-18df-49b8-90da-0c9d278390da-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-bqqvc\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:53.709746 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.709626    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-dshm\") pod \"fc-smoke-test-kserve-7f4687897-bqqvc\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:53.709746 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.709647    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4vp2\" (UniqueName: \"kubernetes.io/projected/f3c9c530-18df-49b8-90da-0c9d278390da-kube-api-access-h4vp2\") pod \"fc-smoke-test-kserve-7f4687897-bqqvc\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:53.709746 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.709674    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-bqqvc\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:53.709746 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.709728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-home\") pod \"fc-smoke-test-kserve-7f4687897-bqqvc\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:53.810922 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.810891    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-home\") pod \"fc-smoke-test-kserve-7f4687897-bqqvc\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:53.811085 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.810938    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-bqqvc\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:53.811085 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.810960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c9c530-18df-49b8-90da-0c9d278390da-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-bqqvc\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:53.811203 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.811081    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-dshm\") pod \"fc-smoke-test-kserve-7f4687897-bqqvc\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:53.811203 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.811130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4vp2\" (UniqueName: \"kubernetes.io/projected/f3c9c530-18df-49b8-90da-0c9d278390da-kube-api-access-h4vp2\") pod \"fc-smoke-test-kserve-7f4687897-bqqvc\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:53.811321 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.811218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-bqqvc\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:53.811380 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.811339    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-bqqvc\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:53.811442 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.811398    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-home\") pod \"fc-smoke-test-kserve-7f4687897-bqqvc\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:53.811546 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.811528    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-bqqvc\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:53.813274 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.813242    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-dshm\") pod \"fc-smoke-test-kserve-7f4687897-bqqvc\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:53.813406 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.813391    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c9c530-18df-49b8-90da-0c9d278390da-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-bqqvc\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:53.819221 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.819198    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4vp2\" (UniqueName: \"kubernetes.io/projected/f3c9c530-18df-49b8-90da-0c9d278390da-kube-api-access-h4vp2\") pod \"fc-smoke-test-kserve-7f4687897-bqqvc\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:53.900836 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:53.900761    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:13:54.033243 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:54.033214    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"]
Jul 27 17:13:54.034931 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:13:54.034903    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c9c530_18df_49b8_90da_0c9d278390da.slice/crio-3359c3212b2f8b3aa4d255b92c14b7699e27c9c28238932bcd22d814888b2988 WatchSource:0}: Error finding container 3359c3212b2f8b3aa4d255b92c14b7699e27c9c28238932bcd22d814888b2988: Status 404 returned error can't find the container with id 3359c3212b2f8b3aa4d255b92c14b7699e27c9c28238932bcd22d814888b2988
Jul 27 17:13:54.155049 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:54.154971    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" event={"ID":"f3c9c530-18df-49b8-90da-0c9d278390da","Type":"ContainerStarted","Data":"3359c3212b2f8b3aa4d255b92c14b7699e27c9c28238932bcd22d814888b2988"}
Jul 27 17:13:56.169179 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:56.169141    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" event={"ID":"f3c9c530-18df-49b8-90da-0c9d278390da","Type":"ContainerStarted","Data":"6d2840ef35efd8c63c96255b8e4c07e1c478deea4a74630cc94321c519e872c9"}
Jul 27 17:13:56.186935 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:56.186886    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" podStartSLOduration=1.199776345 podStartE2EDuration="3.186870262s" totalImagesPullingTime="1.987093917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:13:56.184733097 +0000 UTC m=+780.259997586" watchObservedRunningTime="2026-07-27 17:13:56.186870262 +0000 UTC m=+780.262134748"
Jul 27 17:13:56.900905 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:13:56.900872    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:14:03.210660 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:03.210629    2566 generic.go:356] "Generic (PLEG): container finished" podID="5854c1ae-85a1-4d4f-8d45-792d72188f40" containerID="7ad1501a252bd0dc22b83afde524c6a7832788a621871b380715a8a8576e490e" exitCode=0
Jul 27 17:14:03.211017 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:03.210700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c" event={"ID":"5854c1ae-85a1-4d4f-8d45-792d72188f40","Type":"ContainerDied","Data":"7ad1501a252bd0dc22b83afde524c6a7832788a621871b380715a8a8576e490e"}
Jul 27 17:14:03.211198 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:03.211158    2566 scope.go:117] "RemoveContainer" containerID="7ad1501a252bd0dc22b83afde524c6a7832788a621871b380715a8a8576e490e"
Jul 27 17:14:03.212092 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:03.212065    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 17:14:03.212162 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:03.212134    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 17:14:03.212263 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:03.212245    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 27 17:14:03.901629 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:03.901594    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:14:03.914209 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:03.914158    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:14:04.217484 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:04.217453    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c" event={"ID":"5854c1ae-85a1-4d4f-8d45-792d72188f40","Type":"ContainerStarted","Data":"7eb3258e655190a10df10f5ca8e6102ea598af41af289ce78484b8d45aeffbaf"}
Jul 27 17:14:04.245731 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:04.245697    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:14:04.339192 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:04.332484    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:14:04.339192 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:04.333445    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 27 17:14:04.339192 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:04.333495    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c" podUID="5854c1ae-85a1-4d4f-8d45-792d72188f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 27 17:14:05.329300 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:05.329244    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 27 17:14:05.329930 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:05.329890    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c" podUID="5854c1ae-85a1-4d4f-8d45-792d72188f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 27 17:14:06.329526 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:06.329039    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 27 17:14:06.329526 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:06.329119    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c" podUID="5854c1ae-85a1-4d4f-8d45-792d72188f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 27 17:14:07.328444 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:07.328405    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 27 17:14:07.328743 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:07.328713    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c" podUID="5854c1ae-85a1-4d4f-8d45-792d72188f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 27 17:14:08.329180 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:08.329126    2566 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 27 17:14:08.329651 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:08.329218    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c" podUID="5854c1ae-85a1-4d4f-8d45-792d72188f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 27 17:14:08.680326 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:08.680242    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:14:09.331867 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:09.331832    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:14:09.332725 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:09.332706    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c"
Jul 27 17:14:10.490856 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.490820    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"]
Jul 27 17:14:10.499026 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.499005    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.501697 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.501671    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Jul 27 17:14:10.501821 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.501671    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Jul 27 17:14:10.501821 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.501788    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Jul 27 17:14:10.501821 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.501804    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 27 17:14:10.501962 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.501935    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-np22w\""
Jul 27 17:14:10.507243 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.507221    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"]
Jul 27 17:14:10.595569 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.595540    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23577b52-1aec-4f44-af44-29801987ec61-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.595719 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.595597    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23577b52-1aec-4f44-af44-29801987ec61-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.595719 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.595638    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23577b52-1aec-4f44-af44-29801987ec61-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.595719 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.595673    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23577b52-1aec-4f44-af44-29801987ec61-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.595826 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.595717    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjvzb\" (UniqueName: \"kubernetes.io/projected/23577b52-1aec-4f44-af44-29801987ec61-kube-api-access-qjvzb\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.595826 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.595741    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23577b52-1aec-4f44-af44-29801987ec61-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.595892 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.595821    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23577b52-1aec-4f44-af44-29801987ec61-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.696954 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.696910    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23577b52-1aec-4f44-af44-29801987ec61-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.697151 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.696970    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23577b52-1aec-4f44-af44-29801987ec61-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.697151 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.697009    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23577b52-1aec-4f44-af44-29801987ec61-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.697151 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.697028    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23577b52-1aec-4f44-af44-29801987ec61-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.697151 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.697060    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23577b52-1aec-4f44-af44-29801987ec61-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.697151 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.697083    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjvzb\" (UniqueName: \"kubernetes.io/projected/23577b52-1aec-4f44-af44-29801987ec61-kube-api-access-qjvzb\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.697151 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.697108    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23577b52-1aec-4f44-af44-29801987ec61-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.697453 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.697412    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23577b52-1aec-4f44-af44-29801987ec61-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.697515 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.697477    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23577b52-1aec-4f44-af44-29801987ec61-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.697574 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.697527    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23577b52-1aec-4f44-af44-29801987ec61-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.697574 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.697536    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23577b52-1aec-4f44-af44-29801987ec61-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.699416 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.699392    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23577b52-1aec-4f44-af44-29801987ec61-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.699580 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.699564    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23577b52-1aec-4f44-af44-29801987ec61-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.705501 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.705479    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjvzb\" (UniqueName: \"kubernetes.io/projected/23577b52-1aec-4f44-af44-29801987ec61-kube-api-access-qjvzb\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4\" (UID: \"23577b52-1aec-4f44-af44-29801987ec61\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.810357 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.810272    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:10.960890 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:10.960854    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"]
Jul 27 17:14:10.964652 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:14:10.964615    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23577b52_1aec_4f44_af44_29801987ec61.slice/crio-5ed8b5c5dd21dfe81f5055906d5fd58be0088f35cb821eed63502d67a9b7ce3e WatchSource:0}: Error finding container 5ed8b5c5dd21dfe81f5055906d5fd58be0088f35cb821eed63502d67a9b7ce3e: Status 404 returned error can't find the container with id 5ed8b5c5dd21dfe81f5055906d5fd58be0088f35cb821eed63502d67a9b7ce3e
Jul 27 17:14:11.266434 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:11.266391    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4" event={"ID":"23577b52-1aec-4f44-af44-29801987ec61","Type":"ContainerStarted","Data":"d8b45ad26d0472523c21b3972af4cb01d1ede61de310c0cb08d69dba94e495cb"}
Jul 27 17:14:11.266644 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:11.266459    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4" event={"ID":"23577b52-1aec-4f44-af44-29801987ec61","Type":"ContainerStarted","Data":"5ed8b5c5dd21dfe81f5055906d5fd58be0088f35cb821eed63502d67a9b7ce3e"}
Jul 27 17:14:19.316251 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:19.316219    2566 generic.go:356] "Generic (PLEG): container finished" podID="23577b52-1aec-4f44-af44-29801987ec61" containerID="d8b45ad26d0472523c21b3972af4cb01d1ede61de310c0cb08d69dba94e495cb" exitCode=0
Jul 27 17:14:19.316656 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:19.316257    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4" event={"ID":"23577b52-1aec-4f44-af44-29801987ec61","Type":"ContainerDied","Data":"d8b45ad26d0472523c21b3972af4cb01d1ede61de310c0cb08d69dba94e495cb"}
Jul 27 17:14:36.082064 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:36.082021    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"]
Jul 27 17:14:36.092234 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:14:36.092195    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:14:36.092234 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:14:36.092234 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:14:36.092234 ip-10-0-143-86 kubenswrapper[2566]:  > containerID="6d2840ef35efd8c63c96255b8e4c07e1c478deea4a74630cc94321c519e872c9" cmd=["/bin/sleep","15"]
Jul 27 17:14:36.092562 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:36.092263    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" message="" err=<
Jul 27 17:14:36.092562 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:14:36.092562 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:14:36.092562 ip-10-0-143-86 kubenswrapper[2566]:  >
Jul 27 17:14:36.092562 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:14:36.092282    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:14:36.092562 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:14:36.092562 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:14:36.092562 ip-10-0-143-86 kubenswrapper[2566]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" podUID="f3c9c530-18df-49b8-90da-0c9d278390da" containerName="main" containerID="cri-o://6d2840ef35efd8c63c96255b8e4c07e1c478deea4a74630cc94321c519e872c9"
Jul 27 17:14:36.092562 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:36.092330    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" podUID="f3c9c530-18df-49b8-90da-0c9d278390da" containerName="main" containerID="cri-o://6d2840ef35efd8c63c96255b8e4c07e1c478deea4a74630cc94321c519e872c9" gracePeriod=60
Jul 27 17:14:36.218507 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:36.218457    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" podUID="f3c9c530-18df-49b8-90da-0c9d278390da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 27 17:14:37.218415 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:37.218373    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" podUID="f3c9c530-18df-49b8-90da-0c9d278390da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 27 17:14:37.218899 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:37.218495    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:14:38.218259 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:38.218032    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" podUID="f3c9c530-18df-49b8-90da-0c9d278390da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 27 17:14:39.219027 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:39.218978    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" podUID="f3c9c530-18df-49b8-90da-0c9d278390da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 27 17:14:40.218540 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:40.218491    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" podUID="f3c9c530-18df-49b8-90da-0c9d278390da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 27 17:14:41.218082 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:41.218036    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" podUID="f3c9c530-18df-49b8-90da-0c9d278390da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 27 17:14:42.218902 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:42.218853    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" podUID="f3c9c530-18df-49b8-90da-0c9d278390da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 27 17:14:43.218291 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:43.218245    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" podUID="f3c9c530-18df-49b8-90da-0c9d278390da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 27 17:14:44.218142 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:44.218084    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" podUID="f3c9c530-18df-49b8-90da-0c9d278390da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 27 17:14:45.218302 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:45.218255    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" podUID="f3c9c530-18df-49b8-90da-0c9d278390da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 27 17:14:46.218615 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:46.218563    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" podUID="f3c9c530-18df-49b8-90da-0c9d278390da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 27 17:14:47.218817 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:47.218778    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" podUID="f3c9c530-18df-49b8-90da-0c9d278390da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Jul 27 17:14:47.513604 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:47.513563    2566 generic.go:356] "Generic (PLEG): container finished" podID="f3c9c530-18df-49b8-90da-0c9d278390da" containerID="6d2840ef35efd8c63c96255b8e4c07e1c478deea4a74630cc94321c519e872c9" exitCode=0
Jul 27 17:14:47.513604 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:47.513604    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" event={"ID":"f3c9c530-18df-49b8-90da-0c9d278390da","Type":"ContainerDied","Data":"6d2840ef35efd8c63c96255b8e4c07e1c478deea4a74630cc94321c519e872c9"}
Jul 27 17:14:49.066942 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.066917    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:14:49.166973 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.166942    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c9c530-18df-49b8-90da-0c9d278390da-tls-certs\") pod \"f3c9c530-18df-49b8-90da-0c9d278390da\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") "
Jul 27 17:14:49.166973 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.166976    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4vp2\" (UniqueName: \"kubernetes.io/projected/f3c9c530-18df-49b8-90da-0c9d278390da-kube-api-access-h4vp2\") pod \"f3c9c530-18df-49b8-90da-0c9d278390da\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") "
Jul 27 17:14:49.167223 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.167016    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-model-cache\") pod \"f3c9c530-18df-49b8-90da-0c9d278390da\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") "
Jul 27 17:14:49.167223 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.167033    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-home\") pod \"f3c9c530-18df-49b8-90da-0c9d278390da\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") "
Jul 27 17:14:49.167223 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.167105    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-dshm\") pod \"f3c9c530-18df-49b8-90da-0c9d278390da\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") "
Jul 27 17:14:49.167223 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.167125    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-tmp-dir\") pod \"f3c9c530-18df-49b8-90da-0c9d278390da\" (UID: \"f3c9c530-18df-49b8-90da-0c9d278390da\") "
Jul 27 17:14:49.167452 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.167273    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3c9c530-18df-49b8-90da-0c9d278390da" (UID: "f3c9c530-18df-49b8-90da-0c9d278390da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:14:49.167452 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.167321    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-home" (OuterVolumeSpecName: "home") pod "f3c9c530-18df-49b8-90da-0c9d278390da" (UID: "f3c9c530-18df-49b8-90da-0c9d278390da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:14:49.167452 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.167387    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3c9c530-18df-49b8-90da-0c9d278390da" (UID: "f3c9c530-18df-49b8-90da-0c9d278390da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:14:49.167819 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.167471    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-tmp-dir\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:14:49.167819 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.167485    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-model-cache\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:14:49.167819 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.167496    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-home\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:14:49.169271 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.169244    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c9c530-18df-49b8-90da-0c9d278390da-kube-api-access-h4vp2" (OuterVolumeSpecName: "kube-api-access-h4vp2") pod "f3c9c530-18df-49b8-90da-0c9d278390da" (UID: "f3c9c530-18df-49b8-90da-0c9d278390da"). InnerVolumeSpecName "kube-api-access-h4vp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:14:49.169271 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.169260    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3c9c530-18df-49b8-90da-0c9d278390da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3c9c530-18df-49b8-90da-0c9d278390da" (UID: "f3c9c530-18df-49b8-90da-0c9d278390da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:14:49.169404 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.169245    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-dshm" (OuterVolumeSpecName: "dshm") pod "f3c9c530-18df-49b8-90da-0c9d278390da" (UID: "f3c9c530-18df-49b8-90da-0c9d278390da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:14:49.218406 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.218360    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" podUID="f3c9c530-18df-49b8-90da-0c9d278390da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 27 17:14:49.268132 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.268108    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c9c530-18df-49b8-90da-0c9d278390da-dshm\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:14:49.268132 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.268130    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c9c530-18df-49b8-90da-0c9d278390da-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:14:49.268288 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.268141    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4vp2\" (UniqueName: \"kubernetes.io/projected/f3c9c530-18df-49b8-90da-0c9d278390da-kube-api-access-h4vp2\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:14:49.525801 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.525713    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"
Jul 27 17:14:49.525801 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.525730    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc" event={"ID":"f3c9c530-18df-49b8-90da-0c9d278390da","Type":"ContainerDied","Data":"3359c3212b2f8b3aa4d255b92c14b7699e27c9c28238932bcd22d814888b2988"}
Jul 27 17:14:49.525801 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.525792    2566 scope.go:117] "RemoveContainer" containerID="6d2840ef35efd8c63c96255b8e4c07e1c478deea4a74630cc94321c519e872c9"
Jul 27 17:14:49.553048 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.553020    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"]
Jul 27 17:14:49.557817 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:49.557793    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-bqqvc"]
Jul 27 17:14:50.402767 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:50.402734    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3c9c530-18df-49b8-90da-0c9d278390da" path="/var/lib/kubelet/pods/f3c9c530-18df-49b8-90da-0c9d278390da/volumes"
Jul 27 17:14:50.533196 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:50.533145    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4" event={"ID":"23577b52-1aec-4f44-af44-29801987ec61","Type":"ContainerStarted","Data":"7e9ac095b7b6bbe8b475897af48f45056a66cf764eff352c2b83503723c40474"}
Jul 27 17:14:50.555453 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:50.555395    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4" podStartSLOduration=2.933495625 podStartE2EDuration="40.555377073s" totalImagesPullingTime="30.206658989s" totalInitContainerRuntime="7.415222459s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:14:50.553931065 +0000 UTC m=+834.629195545" watchObservedRunningTime="2026-07-27 17:14:50.555377073 +0000 UTC m=+834.630641561"
Jul 27 17:14:50.810792 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:50.810757    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:50.811004 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:50.810904    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:14:50.812494 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:14:50.812461    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4" podUID="23577b52-1aec-4f44-af44-29801987ec61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 27 17:15:00.810771 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:15:00.810724    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4" podUID="23577b52-1aec-4f44-af44-29801987ec61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 27 17:15:10.810886 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:15:10.810841    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4" podUID="23577b52-1aec-4f44-af44-29801987ec61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 27 17:15:20.810999 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:15:20.810956    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4" podUID="23577b52-1aec-4f44-af44-29801987ec61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 27 17:15:30.810745 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:15:30.810698    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4" podUID="23577b52-1aec-4f44-af44-29801987ec61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 27 17:15:40.811251 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:15:40.811198    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4" podUID="23577b52-1aec-4f44-af44-29801987ec61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 27 17:15:50.811216 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:15:50.811114    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4" podUID="23577b52-1aec-4f44-af44-29801987ec61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 27 17:15:57.343537 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:15:57.343505    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:15:57.345068 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:15:57.345048    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:16:00.811658 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:00.811613    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4" podUID="23577b52-1aec-4f44-af44-29801987ec61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 27 17:16:10.810824 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:10.810761    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4" podUID="23577b52-1aec-4f44-af44-29801987ec61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 27 17:16:12.289914 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.289867    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"]
Jul 27 17:16:12.290454 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.290271    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c9c530-18df-49b8-90da-0c9d278390da" containerName="main"
Jul 27 17:16:12.290454 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.290283    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c9c530-18df-49b8-90da-0c9d278390da" containerName="main"
Jul 27 17:16:12.290454 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.290351    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c9c530-18df-49b8-90da-0c9d278390da" containerName="main"
Jul 27 17:16:12.294696 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.294666    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.299184 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.299134    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-wwsgk\""
Jul 27 17:16:12.299770 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.299741    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 27 17:16:12.300189 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.300152    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Jul 27 17:16:12.300189 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.300159    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Jul 27 17:16:12.303396 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.303371    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"]
Jul 27 17:16:12.321303 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.321274    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.321471 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.321307    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.321471 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.321410    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.321556 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.321495    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.321556 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.321529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58f49\" (UniqueName: \"kubernetes.io/projected/adad9df3-dd8e-4309-b958-74a7dbe738ad-kube-api-access-58f49\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.321637 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.321564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adad9df3-dd8e-4309-b958-74a7dbe738ad-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.422217 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.422151    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.422410 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.422253    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.422410 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.422279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58f49\" (UniqueName: \"kubernetes.io/projected/adad9df3-dd8e-4309-b958-74a7dbe738ad-kube-api-access-58f49\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.422410 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.422301    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adad9df3-dd8e-4309-b958-74a7dbe738ad-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.422410 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.422345    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.422627 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.422454    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.422627 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.422537    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.422732 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.422628    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.422880 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.422859    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.424680 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.424659    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.424741 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.424708    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adad9df3-dd8e-4309-b958-74a7dbe738ad-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.431603 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.431574    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58f49\" (UniqueName: \"kubernetes.io/projected/adad9df3-dd8e-4309-b958-74a7dbe738ad-kube-api-access-58f49\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.610011 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.609926    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:12.755354 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:12.755326    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"]
Jul 27 17:16:12.757196 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:16:12.757136    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadad9df3_dd8e_4309_b958_74a7dbe738ad.slice/crio-f20f6d425fbf41f19380e3c8efa834d5f3aca7285fa5e730e106298b1aaa952b WatchSource:0}: Error finding container f20f6d425fbf41f19380e3c8efa834d5f3aca7285fa5e730e106298b1aaa952b: Status 404 returned error can't find the container with id f20f6d425fbf41f19380e3c8efa834d5f3aca7285fa5e730e106298b1aaa952b
Jul 27 17:16:13.016428 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:13.016386    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt" event={"ID":"adad9df3-dd8e-4309-b958-74a7dbe738ad","Type":"ContainerStarted","Data":"873f534627acba940b04f03d0dae0a71f352d10d2f9c0366a5005fa7289f39a0"}
Jul 27 17:16:13.016607 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:13.016455    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt" event={"ID":"adad9df3-dd8e-4309-b958-74a7dbe738ad","Type":"ContainerStarted","Data":"f20f6d425fbf41f19380e3c8efa834d5f3aca7285fa5e730e106298b1aaa952b"}
Jul 27 17:16:13.039273 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:13.039213    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt" podStartSLOduration=1.039197824 podStartE2EDuration="1.039197824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:16:13.038008736 +0000 UTC m=+917.113273226" watchObservedRunningTime="2026-07-27 17:16:13.039197824 +0000 UTC m=+917.114462308"
Jul 27 17:16:13.610587 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:13.610553    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:20.296553 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.296513    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"]
Jul 27 17:16:20.306433 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:16:20.306283    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:16:20.306433 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:16:20.306433 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:16:20.306433 ip-10-0-143-86 kubenswrapper[2566]:  > containerID="873f534627acba940b04f03d0dae0a71f352d10d2f9c0366a5005fa7289f39a0" cmd=["/bin/sleep","15"]
Jul 27 17:16:20.306433 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.306341    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt" message="" err=<
Jul 27 17:16:20.306433 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:16:20.306433 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:16:20.306433 ip-10-0-143-86 kubenswrapper[2566]:  >
Jul 27 17:16:20.306433 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:16:20.306358    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:16:20.306433 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:16:20.306433 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:16:20.306433 ip-10-0-143-86 kubenswrapper[2566]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt" podUID="adad9df3-dd8e-4309-b958-74a7dbe738ad" containerName="main" containerID="cri-o://873f534627acba940b04f03d0dae0a71f352d10d2f9c0366a5005fa7289f39a0"
Jul 27 17:16:20.306433 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.306393    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt" podUID="adad9df3-dd8e-4309-b958-74a7dbe738ad" containerName="main" containerID="cri-o://873f534627acba940b04f03d0dae0a71f352d10d2f9c0366a5005fa7289f39a0" gracePeriod=60
Jul 27 17:16:20.656623 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.656593    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:20.700475 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.700443    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58f49\" (UniqueName: \"kubernetes.io/projected/adad9df3-dd8e-4309-b958-74a7dbe738ad-kube-api-access-58f49\") pod \"adad9df3-dd8e-4309-b958-74a7dbe738ad\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") "
Jul 27 17:16:20.700653 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.700486    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adad9df3-dd8e-4309-b958-74a7dbe738ad-tls-certs\") pod \"adad9df3-dd8e-4309-b958-74a7dbe738ad\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") "
Jul 27 17:16:20.700653 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.700507    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-model-cache\") pod \"adad9df3-dd8e-4309-b958-74a7dbe738ad\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") "
Jul 27 17:16:20.700653 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.700534    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-dshm\") pod \"adad9df3-dd8e-4309-b958-74a7dbe738ad\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") "
Jul 27 17:16:20.700653 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.700622    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-home\") pod \"adad9df3-dd8e-4309-b958-74a7dbe738ad\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") "
Jul 27 17:16:20.700872 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.700658    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-tmp-dir\") pod \"adad9df3-dd8e-4309-b958-74a7dbe738ad\" (UID: \"adad9df3-dd8e-4309-b958-74a7dbe738ad\") "
Jul 27 17:16:20.700872 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.700743    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "adad9df3-dd8e-4309-b958-74a7dbe738ad" (UID: "adad9df3-dd8e-4309-b958-74a7dbe738ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:16:20.700983 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.700917    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-home" (OuterVolumeSpecName: "home") pod "adad9df3-dd8e-4309-b958-74a7dbe738ad" (UID: "adad9df3-dd8e-4309-b958-74a7dbe738ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:16:20.701034 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.700991    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "adad9df3-dd8e-4309-b958-74a7dbe738ad" (UID: "adad9df3-dd8e-4309-b958-74a7dbe738ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:16:20.701092 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.701053    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-model-cache\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:16:20.701092 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.701070    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-home\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:16:20.702748 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.702720    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-dshm" (OuterVolumeSpecName: "dshm") pod "adad9df3-dd8e-4309-b958-74a7dbe738ad" (UID: "adad9df3-dd8e-4309-b958-74a7dbe738ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:16:20.702868 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.702818    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adad9df3-dd8e-4309-b958-74a7dbe738ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "adad9df3-dd8e-4309-b958-74a7dbe738ad" (UID: "adad9df3-dd8e-4309-b958-74a7dbe738ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:16:20.702918 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.702870    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adad9df3-dd8e-4309-b958-74a7dbe738ad-kube-api-access-58f49" (OuterVolumeSpecName: "kube-api-access-58f49") pod "adad9df3-dd8e-4309-b958-74a7dbe738ad" (UID: "adad9df3-dd8e-4309-b958-74a7dbe738ad"). InnerVolumeSpecName "kube-api-access-58f49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:16:20.802337 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.802302    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58f49\" (UniqueName: \"kubernetes.io/projected/adad9df3-dd8e-4309-b958-74a7dbe738ad-kube-api-access-58f49\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:16:20.802337 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.802330    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adad9df3-dd8e-4309-b958-74a7dbe738ad-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:16:20.802337 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.802341    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-dshm\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:16:20.802337 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.802349    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adad9df3-dd8e-4309-b958-74a7dbe738ad-tmp-dir\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:16:20.811443 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:20.811414    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4" podUID="23577b52-1aec-4f44-af44-29801987ec61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 27 17:16:21.066320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:21.066281    2566 generic.go:356] "Generic (PLEG): container finished" podID="adad9df3-dd8e-4309-b958-74a7dbe738ad" containerID="873f534627acba940b04f03d0dae0a71f352d10d2f9c0366a5005fa7289f39a0" exitCode=0
Jul 27 17:16:21.066320 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:21.066320    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"
Jul 27 17:16:21.066535 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:21.066366    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt" event={"ID":"adad9df3-dd8e-4309-b958-74a7dbe738ad","Type":"ContainerDied","Data":"873f534627acba940b04f03d0dae0a71f352d10d2f9c0366a5005fa7289f39a0"}
Jul 27 17:16:21.066535 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:21.066418    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt" event={"ID":"adad9df3-dd8e-4309-b958-74a7dbe738ad","Type":"ContainerDied","Data":"f20f6d425fbf41f19380e3c8efa834d5f3aca7285fa5e730e106298b1aaa952b"}
Jul 27 17:16:21.066535 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:21.066441    2566 scope.go:117] "RemoveContainer" containerID="873f534627acba940b04f03d0dae0a71f352d10d2f9c0366a5005fa7289f39a0"
Jul 27 17:16:21.077810 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:21.077792    2566 scope.go:117] "RemoveContainer" containerID="873f534627acba940b04f03d0dae0a71f352d10d2f9c0366a5005fa7289f39a0"
Jul 27 17:16:21.078118 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:16:21.078096    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"873f534627acba940b04f03d0dae0a71f352d10d2f9c0366a5005fa7289f39a0\": container with ID starting with 873f534627acba940b04f03d0dae0a71f352d10d2f9c0366a5005fa7289f39a0 not found: ID does not exist" containerID="873f534627acba940b04f03d0dae0a71f352d10d2f9c0366a5005fa7289f39a0"
Jul 27 17:16:21.078212 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:21.078131    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873f534627acba940b04f03d0dae0a71f352d10d2f9c0366a5005fa7289f39a0"} err="failed to get container status \"873f534627acba940b04f03d0dae0a71f352d10d2f9c0366a5005fa7289f39a0\": rpc error: code = NotFound desc = could not find container \"873f534627acba940b04f03d0dae0a71f352d10d2f9c0366a5005fa7289f39a0\": container with ID starting with 873f534627acba940b04f03d0dae0a71f352d10d2f9c0366a5005fa7289f39a0 not found: ID does not exist"
Jul 27 17:16:21.088411 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:21.088384    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"]
Jul 27 17:16:21.093127 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:21.093091    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gtxqt"]
Jul 27 17:16:22.405217 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:22.405153    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adad9df3-dd8e-4309-b958-74a7dbe738ad" path="/var/lib/kubelet/pods/adad9df3-dd8e-4309-b958-74a7dbe738ad/volumes"
Jul 27 17:16:30.811272 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:30.811221    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4" podUID="23577b52-1aec-4f44-af44-29801987ec61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 27 17:16:39.741730 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.741687    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 27 17:16:39.742342 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.742320    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adad9df3-dd8e-4309-b958-74a7dbe738ad" containerName="main"
Jul 27 17:16:39.742435 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.742344    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="adad9df3-dd8e-4309-b958-74a7dbe738ad" containerName="main"
Jul 27 17:16:39.742492 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.742468    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="adad9df3-dd8e-4309-b958-74a7dbe738ad" containerName="main"
Jul 27 17:16:39.748972 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.748949    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.751597 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.751571    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 27 17:16:39.751722 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.751706    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-cq8pb\""
Jul 27 17:16:39.751799 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.751717    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 27 17:16:39.751799 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.751707    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 27 17:16:39.751894 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.751827    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 27 17:16:39.752525 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.752509    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-tzkd5\""
Jul 27 17:16:39.759536 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.759512    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 27 17:16:39.871074 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.871037    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j6qm\" (UniqueName: \"kubernetes.io/projected/4b88436b-0fc3-46c8-9541-f8f648917cfe-kube-api-access-7j6qm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.871268 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.871080    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.871268 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.871112    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.871268 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.871194    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.871268 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.871222    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.871425 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.871296    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b88436b-0fc3-46c8-9541-f8f648917cfe-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.871425 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.871347    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.972096 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.972057    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b88436b-0fc3-46c8-9541-f8f648917cfe-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.972300 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.972126    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.972300 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.972181    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j6qm\" (UniqueName: \"kubernetes.io/projected/4b88436b-0fc3-46c8-9541-f8f648917cfe-kube-api-access-7j6qm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.972300 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.972238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.972300 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.972286    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.972533 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.972321    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.972533 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.972345    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.972666 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.972631    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.972723 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.972695    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.972763 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.972724    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.972952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.972925    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.974617 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.974587    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.974705 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.974631    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b88436b-0fc3-46c8-9541-f8f648917cfe-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:39.979900 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:39.979875    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j6qm\" (UniqueName: \"kubernetes.io/projected/4b88436b-0fc3-46c8-9541-f8f648917cfe-kube-api-access-7j6qm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:40.061139 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:40.061053    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:16:40.212455 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:16:40.212424    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b88436b_0fc3_46c8_9541_f8f648917cfe.slice/crio-c00a0b2aa2ee99ab8869bfa928293b97dc3e57ec9b465e7ecb0148a2921242d0 WatchSource:0}: Error finding container c00a0b2aa2ee99ab8869bfa928293b97dc3e57ec9b465e7ecb0148a2921242d0: Status 404 returned error can't find the container with id c00a0b2aa2ee99ab8869bfa928293b97dc3e57ec9b465e7ecb0148a2921242d0
Jul 27 17:16:40.212659 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:40.212631    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 27 17:16:40.829155 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:40.829112    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:16:40.843246 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:40.843208    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59dc67d6c58mpn4"
Jul 27 17:16:41.189534 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:41.189451    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"4b88436b-0fc3-46c8-9541-f8f648917cfe","Type":"ContainerStarted","Data":"a8402d91788a4a4f42c993f25917a45e5b7db798630c6c9a1905b6075edd9e85"}
Jul 27 17:16:41.189678 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:41.189536    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"4b88436b-0fc3-46c8-9541-f8f648917cfe","Type":"ContainerStarted","Data":"c00a0b2aa2ee99ab8869bfa928293b97dc3e57ec9b465e7ecb0148a2921242d0"}
Jul 27 17:16:50.244482 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:50.244447    2566 generic.go:356] "Generic (PLEG): container finished" podID="4b88436b-0fc3-46c8-9541-f8f648917cfe" containerID="a8402d91788a4a4f42c993f25917a45e5b7db798630c6c9a1905b6075edd9e85" exitCode=0
Jul 27 17:16:50.244881 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:50.244506    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"4b88436b-0fc3-46c8-9541-f8f648917cfe","Type":"ContainerDied","Data":"a8402d91788a4a4f42c993f25917a45e5b7db798630c6c9a1905b6075edd9e85"}
Jul 27 17:16:51.251708 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:51.251669    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"4b88436b-0fc3-46c8-9541-f8f648917cfe","Type":"ContainerStarted","Data":"11f7abf495932e32b2f4df285b24ec37c5e0af96c957180e289359371194b0a2"}
Jul 27 17:16:51.274769 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:51.274712    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.957086831 podStartE2EDuration="12.274697485s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.317610654s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:16:51.271680584 +0000 UTC m=+955.346945071" watchObservedRunningTime="2026-07-27 17:16:51.274697485 +0000 UTC m=+955.349961972"
Jul 27 17:16:52.061366 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:16:52.061320    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:17:00.061408 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:17:00.061362    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:17:00.063037 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:17:00.063007    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b88436b-0fc3-46c8-9541-f8f648917cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 27 17:17:10.061696 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:17:10.061653    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b88436b-0fc3-46c8-9541-f8f648917cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 27 17:17:20.062225 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:17:20.062113    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b88436b-0fc3-46c8-9541-f8f648917cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 27 17:17:30.062274 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:17:30.062218    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b88436b-0fc3-46c8-9541-f8f648917cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 27 17:17:40.062076 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:17:40.062025    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b88436b-0fc3-46c8-9541-f8f648917cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 27 17:17:50.062379 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:17:50.062319    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b88436b-0fc3-46c8-9541-f8f648917cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 27 17:18:00.061616 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:18:00.061562    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b88436b-0fc3-46c8-9541-f8f648917cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 27 17:18:10.061861 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:18:10.061816    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b88436b-0fc3-46c8-9541-f8f648917cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 27 17:18:20.062310 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:18:20.062259    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b88436b-0fc3-46c8-9541-f8f648917cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 27 17:18:30.071337 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:18:30.071302    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:18:30.079084 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:18:30.079046    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:20:01.839150 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:01.839113    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 27 17:20:16.861456 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:16.861394    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b88436b-0fc3-46c8-9541-f8f648917cfe" containerName="main" containerID="cri-o://11f7abf495932e32b2f4df285b24ec37c5e0af96c957180e289359371194b0a2" gracePeriod=45
Jul 27 17:20:17.072066 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.072012    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b88436b-0fc3-46c8-9541-f8f648917cfe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Jul 27 17:20:17.420688 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.420657    2566 generic.go:356] "Generic (PLEG): container finished" podID="4b88436b-0fc3-46c8-9541-f8f648917cfe" containerID="11f7abf495932e32b2f4df285b24ec37c5e0af96c957180e289359371194b0a2" exitCode=0
Jul 27 17:20:17.420836 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.420709    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"4b88436b-0fc3-46c8-9541-f8f648917cfe","Type":"ContainerDied","Data":"11f7abf495932e32b2f4df285b24ec37c5e0af96c957180e289359371194b0a2"}
Jul 27 17:20:17.541841 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.541818    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:20:17.561424 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.561393    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j6qm\" (UniqueName: \"kubernetes.io/projected/4b88436b-0fc3-46c8-9541-f8f648917cfe-kube-api-access-7j6qm\") pod \"4b88436b-0fc3-46c8-9541-f8f648917cfe\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") "
Jul 27 17:20:17.561579 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.561444    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-model-cache\") pod \"4b88436b-0fc3-46c8-9541-f8f648917cfe\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") "
Jul 27 17:20:17.561579 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.561500    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-tmp-dir\") pod \"4b88436b-0fc3-46c8-9541-f8f648917cfe\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") "
Jul 27 17:20:17.561579 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.561567    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-kserve-provision-location\") pod \"4b88436b-0fc3-46c8-9541-f8f648917cfe\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") "
Jul 27 17:20:17.561748 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.561588    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-home\") pod \"4b88436b-0fc3-46c8-9541-f8f648917cfe\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") "
Jul 27 17:20:17.561748 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.561629    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-dshm\") pod \"4b88436b-0fc3-46c8-9541-f8f648917cfe\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") "
Jul 27 17:20:17.561748 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.561657    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b88436b-0fc3-46c8-9541-f8f648917cfe-tls-certs\") pod \"4b88436b-0fc3-46c8-9541-f8f648917cfe\" (UID: \"4b88436b-0fc3-46c8-9541-f8f648917cfe\") "
Jul 27 17:20:17.563005 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.561740    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b88436b-0fc3-46c8-9541-f8f648917cfe" (UID: "4b88436b-0fc3-46c8-9541-f8f648917cfe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:20:17.563005 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.562377    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-model-cache\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:20:17.563005 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.562466    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-home" (OuterVolumeSpecName: "home") pod "4b88436b-0fc3-46c8-9541-f8f648917cfe" (UID: "4b88436b-0fc3-46c8-9541-f8f648917cfe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:20:17.564720 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.564383    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b88436b-0fc3-46c8-9541-f8f648917cfe-kube-api-access-7j6qm" (OuterVolumeSpecName: "kube-api-access-7j6qm") pod "4b88436b-0fc3-46c8-9541-f8f648917cfe" (UID: "4b88436b-0fc3-46c8-9541-f8f648917cfe"). InnerVolumeSpecName "kube-api-access-7j6qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:20:17.565419 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.565392    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b88436b-0fc3-46c8-9541-f8f648917cfe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b88436b-0fc3-46c8-9541-f8f648917cfe" (UID: "4b88436b-0fc3-46c8-9541-f8f648917cfe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:20:17.572258 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.571335    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-dshm" (OuterVolumeSpecName: "dshm") pod "4b88436b-0fc3-46c8-9541-f8f648917cfe" (UID: "4b88436b-0fc3-46c8-9541-f8f648917cfe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:20:17.589626 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.589589    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b88436b-0fc3-46c8-9541-f8f648917cfe" (UID: "4b88436b-0fc3-46c8-9541-f8f648917cfe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:20:17.624248 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.624163    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b88436b-0fc3-46c8-9541-f8f648917cfe" (UID: "4b88436b-0fc3-46c8-9541-f8f648917cfe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:20:17.663071 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.663040    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-tmp-dir\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:20:17.663071 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.663069    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-kserve-provision-location\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:20:17.663255 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.663084    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-home\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:20:17.663255 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.663096    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b88436b-0fc3-46c8-9541-f8f648917cfe-dshm\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:20:17.663255 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.663107    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b88436b-0fc3-46c8-9541-f8f648917cfe-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:20:17.663255 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:17.663118    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j6qm\" (UniqueName: \"kubernetes.io/projected/4b88436b-0fc3-46c8-9541-f8f648917cfe-kube-api-access-7j6qm\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:20:18.429233 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:18.429196    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"4b88436b-0fc3-46c8-9541-f8f648917cfe","Type":"ContainerDied","Data":"c00a0b2aa2ee99ab8869bfa928293b97dc3e57ec9b465e7ecb0148a2921242d0"}
Jul 27 17:20:18.429637 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:18.429242    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 27 17:20:18.429637 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:18.429268    2566 scope.go:117] "RemoveContainer" containerID="11f7abf495932e32b2f4df285b24ec37c5e0af96c957180e289359371194b0a2"
Jul 27 17:20:18.438788 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:18.438768    2566 scope.go:117] "RemoveContainer" containerID="a8402d91788a4a4f42c993f25917a45e5b7db798630c6c9a1905b6075edd9e85"
Jul 27 17:20:18.448715 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:18.448694    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 27 17:20:18.454695 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:18.454670    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 27 17:20:20.403532 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:20.403497    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b88436b-0fc3-46c8-9541-f8f648917cfe" path="/var/lib/kubelet/pods/4b88436b-0fc3-46c8-9541-f8f648917cfe/volumes"
Jul 27 17:20:31.285958 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.285920    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"]
Jul 27 17:20:31.286462 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.286327    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b88436b-0fc3-46c8-9541-f8f648917cfe" containerName="main"
Jul 27 17:20:31.286462 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.286339    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b88436b-0fc3-46c8-9541-f8f648917cfe" containerName="main"
Jul 27 17:20:31.286462 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.286347    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b88436b-0fc3-46c8-9541-f8f648917cfe" containerName="storage-initializer"
Jul 27 17:20:31.286462 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.286353    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b88436b-0fc3-46c8-9541-f8f648917cfe" containerName="storage-initializer"
Jul 27 17:20:31.286462 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.286415    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b88436b-0fc3-46c8-9541-f8f648917cfe" containerName="main"
Jul 27 17:20:31.289566 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.289547    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.292131 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.292107    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-7ptbw\""
Jul 27 17:20:31.292301 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.292278    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 27 17:20:31.292406 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.292322    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Jul 27 17:20:31.292406 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.292366    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Jul 27 17:20:31.293105 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.293087    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Jul 27 17:20:31.306864 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.306838    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"]
Jul 27 17:20:31.384903 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.384866    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.385095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.384923    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-dshm\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.385095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.384972    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvmqz\" (UniqueName: \"kubernetes.io/projected/c6702df0-6af0-4de7-bc40-84d7e1da0042-kube-api-access-tvmqz\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.385095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.385001    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-home\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.385095 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.385027    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6702df0-6af0-4de7-bc40-84d7e1da0042-tls-certs\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.385333 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.385123    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-tmp-dir\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.385333 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.385162    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-model-cache\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.486293 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.486262    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.486453 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.486311    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-dshm\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.486453 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.486339    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvmqz\" (UniqueName: \"kubernetes.io/projected/c6702df0-6af0-4de7-bc40-84d7e1da0042-kube-api-access-tvmqz\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.486453 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.486365    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-home\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.486453 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.486384    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6702df0-6af0-4de7-bc40-84d7e1da0042-tls-certs\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.486453 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.486417    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-tmp-dir\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.486453 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.486442    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-model-cache\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.486777 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.486699    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.486842 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.486797    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-home\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.486904 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.486881    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-model-cache\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.486904 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.486889    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-tmp-dir\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.488588 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.488569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-dshm\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.488853 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.488834    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6702df0-6af0-4de7-bc40-84d7e1da0042-tls-certs\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.494592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.494570    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvmqz\" (UniqueName: \"kubernetes.io/projected/c6702df0-6af0-4de7-bc40-84d7e1da0042-kube-api-access-tvmqz\") pod \"auth-invalid-token-test-kserve-5f86979cb9-cm7f6\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.599787 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.599708    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:31.741077 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:31.741043    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"]
Jul 27 17:20:31.742364 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:20:31.742336    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6702df0_6af0_4de7_bc40_84d7e1da0042.slice/crio-ca91bef9a82a44d6c816a1792909df8ea54e59c0ee0dc8aeba7b85b41736727b WatchSource:0}: Error finding container ca91bef9a82a44d6c816a1792909df8ea54e59c0ee0dc8aeba7b85b41736727b: Status 404 returned error can't find the container with id ca91bef9a82a44d6c816a1792909df8ea54e59c0ee0dc8aeba7b85b41736727b
Jul 27 17:20:32.507110 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:32.507074    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" event={"ID":"c6702df0-6af0-4de7-bc40-84d7e1da0042","Type":"ContainerStarted","Data":"1147697f7c52657bcbd84985c90af1734647aa70106188e345c1a0977dec7a9a"}
Jul 27 17:20:32.507567 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:32.507132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" event={"ID":"c6702df0-6af0-4de7-bc40-84d7e1da0042","Type":"ContainerStarted","Data":"ca91bef9a82a44d6c816a1792909df8ea54e59c0ee0dc8aeba7b85b41736727b"}
Jul 27 17:20:39.552091 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:39.552056    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerID="1147697f7c52657bcbd84985c90af1734647aa70106188e345c1a0977dec7a9a" exitCode=0
Jul 27 17:20:39.552803 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:39.552138    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" event={"ID":"c6702df0-6af0-4de7-bc40-84d7e1da0042","Type":"ContainerDied","Data":"1147697f7c52657bcbd84985c90af1734647aa70106188e345c1a0977dec7a9a"}
Jul 27 17:20:40.560707 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:40.560670    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" event={"ID":"c6702df0-6af0-4de7-bc40-84d7e1da0042","Type":"ContainerStarted","Data":"99c479e5fc673cf3d7dcfde40338d91d52a8585c8a1ad1b1d1e06605dd41f096"}
Jul 27 17:20:40.586259 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:40.586204    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" podStartSLOduration=2.372168089 podStartE2EDuration="9.586183699s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.21401561s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:20:40.584459158 +0000 UTC m=+1184.659723644" watchObservedRunningTime="2026-07-27 17:20:40.586183699 +0000 UTC m=+1184.661448187"
Jul 27 17:20:40.600253 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:40.600226    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:41.600547 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:41.600507    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:20:41.602302 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:41.602269    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" podUID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 27 17:20:51.600993 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:51.600947    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" podUID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 27 17:20:57.387013 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:57.386979    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:20:57.388798 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:20:57.388768    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:21:01.600452 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:21:01.600396    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" podUID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 27 17:21:11.600293 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:21:11.600235    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" podUID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 27 17:21:21.600824 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:21:21.600772    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" podUID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 27 17:21:31.600144 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:21:31.600103    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" podUID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 27 17:21:41.600931 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:21:41.600878    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" podUID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 27 17:21:51.600273 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:21:51.600225    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" podUID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 27 17:22:01.600390 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:01.600347    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" podUID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 27 17:22:11.600492 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:11.600450    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" podUID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 27 17:22:21.600367 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:21.600318    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" podUID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 27 17:22:31.611011 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:31.610977    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:22:31.619418 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:31.619385    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:22:33.225836 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:33.225800    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"]
Jul 27 17:22:48.262050 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:48.262004    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" podUID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerName="main" containerID="cri-o://99c479e5fc673cf3d7dcfde40338d91d52a8585c8a1ad1b1d1e06605dd41f096" gracePeriod=45
Jul 27 17:22:48.611779 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:48.611674    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" podUID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Jul 27 17:22:48.927350 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:48.927315    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:22:49.082524 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.082491    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvmqz\" (UniqueName: \"kubernetes.io/projected/c6702df0-6af0-4de7-bc40-84d7e1da0042-kube-api-access-tvmqz\") pod \"c6702df0-6af0-4de7-bc40-84d7e1da0042\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") "
Jul 27 17:22:49.082697 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.082533    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-model-cache\") pod \"c6702df0-6af0-4de7-bc40-84d7e1da0042\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") "
Jul 27 17:22:49.082697 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.082555    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-home\") pod \"c6702df0-6af0-4de7-bc40-84d7e1da0042\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") "
Jul 27 17:22:49.082697 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.082573    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-tmp-dir\") pod \"c6702df0-6af0-4de7-bc40-84d7e1da0042\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") "
Jul 27 17:22:49.082817 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.082787    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-dshm\") pod \"c6702df0-6af0-4de7-bc40-84d7e1da0042\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") "
Jul 27 17:22:49.082865 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.082815    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6702df0-6af0-4de7-bc40-84d7e1da0042" (UID: "c6702df0-6af0-4de7-bc40-84d7e1da0042"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:22:49.082865 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.082830    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6702df0-6af0-4de7-bc40-84d7e1da0042-tls-certs\") pod \"c6702df0-6af0-4de7-bc40-84d7e1da0042\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") "
Jul 27 17:22:49.082982 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.082957    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-kserve-provision-location\") pod \"c6702df0-6af0-4de7-bc40-84d7e1da0042\" (UID: \"c6702df0-6af0-4de7-bc40-84d7e1da0042\") "
Jul 27 17:22:49.083312 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.083289    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-model-cache\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:22:49.083427 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.083376    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-home" (OuterVolumeSpecName: "home") pod "c6702df0-6af0-4de7-bc40-84d7e1da0042" (UID: "c6702df0-6af0-4de7-bc40-84d7e1da0042"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:22:49.085344 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.085316    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-dshm" (OuterVolumeSpecName: "dshm") pod "c6702df0-6af0-4de7-bc40-84d7e1da0042" (UID: "c6702df0-6af0-4de7-bc40-84d7e1da0042"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:22:49.085735 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.085715    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6702df0-6af0-4de7-bc40-84d7e1da0042-kube-api-access-tvmqz" (OuterVolumeSpecName: "kube-api-access-tvmqz") pod "c6702df0-6af0-4de7-bc40-84d7e1da0042" (UID: "c6702df0-6af0-4de7-bc40-84d7e1da0042"). InnerVolumeSpecName "kube-api-access-tvmqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:22:49.085803 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.085714    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6702df0-6af0-4de7-bc40-84d7e1da0042-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6702df0-6af0-4de7-bc40-84d7e1da0042" (UID: "c6702df0-6af0-4de7-bc40-84d7e1da0042"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:22:49.095254 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.095232    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6702df0-6af0-4de7-bc40-84d7e1da0042" (UID: "c6702df0-6af0-4de7-bc40-84d7e1da0042"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:22:49.143298 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.143250    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c6702df0-6af0-4de7-bc40-84d7e1da0042" (UID: "c6702df0-6af0-4de7-bc40-84d7e1da0042"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:22:49.184291 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.184223    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-kserve-provision-location\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:22:49.184291 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.184250    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvmqz\" (UniqueName: \"kubernetes.io/projected/c6702df0-6af0-4de7-bc40-84d7e1da0042-kube-api-access-tvmqz\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:22:49.184291 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.184261    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-home\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:22:49.184291 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.184269    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-tmp-dir\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:22:49.184291 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.184277    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6702df0-6af0-4de7-bc40-84d7e1da0042-dshm\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:22:49.184291 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.184285    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6702df0-6af0-4de7-bc40-84d7e1da0042-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:22:49.330539 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.330498    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerID="99c479e5fc673cf3d7dcfde40338d91d52a8585c8a1ad1b1d1e06605dd41f096" exitCode=0
Jul 27 17:22:49.330952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.330551    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"
Jul 27 17:22:49.330952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.330579    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" event={"ID":"c6702df0-6af0-4de7-bc40-84d7e1da0042","Type":"ContainerDied","Data":"99c479e5fc673cf3d7dcfde40338d91d52a8585c8a1ad1b1d1e06605dd41f096"}
Jul 27 17:22:49.330952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.330637    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6" event={"ID":"c6702df0-6af0-4de7-bc40-84d7e1da0042","Type":"ContainerDied","Data":"ca91bef9a82a44d6c816a1792909df8ea54e59c0ee0dc8aeba7b85b41736727b"}
Jul 27 17:22:49.330952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.330670    2566 scope.go:117] "RemoveContainer" containerID="99c479e5fc673cf3d7dcfde40338d91d52a8585c8a1ad1b1d1e06605dd41f096"
Jul 27 17:22:49.340395 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.340378    2566 scope.go:117] "RemoveContainer" containerID="1147697f7c52657bcbd84985c90af1734647aa70106188e345c1a0977dec7a9a"
Jul 27 17:22:49.348930 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.348914    2566 scope.go:117] "RemoveContainer" containerID="99c479e5fc673cf3d7dcfde40338d91d52a8585c8a1ad1b1d1e06605dd41f096"
Jul 27 17:22:49.349197 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:22:49.349158    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c479e5fc673cf3d7dcfde40338d91d52a8585c8a1ad1b1d1e06605dd41f096\": container with ID starting with 99c479e5fc673cf3d7dcfde40338d91d52a8585c8a1ad1b1d1e06605dd41f096 not found: ID does not exist" containerID="99c479e5fc673cf3d7dcfde40338d91d52a8585c8a1ad1b1d1e06605dd41f096"
Jul 27 17:22:49.349278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.349211    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c479e5fc673cf3d7dcfde40338d91d52a8585c8a1ad1b1d1e06605dd41f096"} err="failed to get container status \"99c479e5fc673cf3d7dcfde40338d91d52a8585c8a1ad1b1d1e06605dd41f096\": rpc error: code = NotFound desc = could not find container \"99c479e5fc673cf3d7dcfde40338d91d52a8585c8a1ad1b1d1e06605dd41f096\": container with ID starting with 99c479e5fc673cf3d7dcfde40338d91d52a8585c8a1ad1b1d1e06605dd41f096 not found: ID does not exist"
Jul 27 17:22:49.349278 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.349234    2566 scope.go:117] "RemoveContainer" containerID="1147697f7c52657bcbd84985c90af1734647aa70106188e345c1a0977dec7a9a"
Jul 27 17:22:49.349499 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:22:49.349480    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1147697f7c52657bcbd84985c90af1734647aa70106188e345c1a0977dec7a9a\": container with ID starting with 1147697f7c52657bcbd84985c90af1734647aa70106188e345c1a0977dec7a9a not found: ID does not exist" containerID="1147697f7c52657bcbd84985c90af1734647aa70106188e345c1a0977dec7a9a"
Jul 27 17:22:49.349545 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.349504    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1147697f7c52657bcbd84985c90af1734647aa70106188e345c1a0977dec7a9a"} err="failed to get container status \"1147697f7c52657bcbd84985c90af1734647aa70106188e345c1a0977dec7a9a\": rpc error: code = NotFound desc = could not find container \"1147697f7c52657bcbd84985c90af1734647aa70106188e345c1a0977dec7a9a\": container with ID starting with 1147697f7c52657bcbd84985c90af1734647aa70106188e345c1a0977dec7a9a not found: ID does not exist"
Jul 27 17:22:49.355965 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.355945    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"]
Jul 27 17:22:49.364447 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:49.364426    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5f86979cb9-cm7f6"]
Jul 27 17:22:50.402904 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:22:50.402871    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6702df0-6af0-4de7-bc40-84d7e1da0042" path="/var/lib/kubelet/pods/c6702df0-6af0-4de7-bc40-84d7e1da0042/volumes"
Jul 27 17:23:12.409135 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.409103    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"]
Jul 27 17:23:12.409546 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.409472    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerName="main"
Jul 27 17:23:12.409546 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.409485    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerName="main"
Jul 27 17:23:12.409546 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.409497    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerName="storage-initializer"
Jul 27 17:23:12.409546 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.409503    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerName="storage-initializer"
Jul 27 17:23:12.409673 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.409561    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6702df0-6af0-4de7-bc40-84d7e1da0042" containerName="main"
Jul 27 17:23:12.413961 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.413943    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.416705 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.416683    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Jul 27 17:23:12.416705 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.416700    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Jul 27 17:23:12.416893 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.416724    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-f9mmq\""
Jul 27 17:23:12.416893 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.416835    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 27 17:23:12.417010 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.416910    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Jul 27 17:23:12.422617 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.422594    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"]
Jul 27 17:23:12.489897 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.489853    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de6f1e1d-729d-40dc-bb32-b5421356a26e-tls-certs\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.489897 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.489901    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-dshm\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.490107 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.489937    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-model-cache\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.490107 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.490049    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl5r9\" (UniqueName: \"kubernetes.io/projected/de6f1e1d-729d-40dc-bb32-b5421356a26e-kube-api-access-jl5r9\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.490107 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.490082    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-home\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.490301 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.490214    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-kserve-provision-location\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.490360 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.490308    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-tmp-dir\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.591332 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.591292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de6f1e1d-729d-40dc-bb32-b5421356a26e-tls-certs\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.591332 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.591329    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-dshm\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.591579 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.591346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-model-cache\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.591579 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.591395    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl5r9\" (UniqueName: \"kubernetes.io/projected/de6f1e1d-729d-40dc-bb32-b5421356a26e-kube-api-access-jl5r9\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.591579 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.591413    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-home\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.591579 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.591443    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-kserve-provision-location\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.591579 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.591473    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-tmp-dir\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.591832 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.591811    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-model-cache\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.591872 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.591841    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-home\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.591920 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.591887    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-tmp-dir\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.591920 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.591901    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-kserve-provision-location\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.593605 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.593586    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-dshm\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.593914 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.593896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de6f1e1d-729d-40dc-bb32-b5421356a26e-tls-certs\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.601292 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.601267    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl5r9\" (UniqueName: \"kubernetes.io/projected/de6f1e1d-729d-40dc-bb32-b5421356a26e-kube-api-access-jl5r9\") pod \"auth-disabled-test-kserve-755f6dd566-d8jsm\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.724979 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.724947    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:12.862065 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:12.862040    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"]
Jul 27 17:23:12.864058 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:23:12.864025    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6f1e1d_729d_40dc_bb32_b5421356a26e.slice/crio-45c31f93e4780a0eea72f2772af2853e25d08c0c3c0945077b4284ec1035bd49 WatchSource:0}: Error finding container 45c31f93e4780a0eea72f2772af2853e25d08c0c3c0945077b4284ec1035bd49: Status 404 returned error can't find the container with id 45c31f93e4780a0eea72f2772af2853e25d08c0c3c0945077b4284ec1035bd49
Jul 27 17:23:13.467729 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:13.467695    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" event={"ID":"de6f1e1d-729d-40dc-bb32-b5421356a26e","Type":"ContainerStarted","Data":"47eb389ad123e1116885985a49c6a687bc22885245a3be7ce1b93b78757f40de"}
Jul 27 17:23:13.468223 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:13.467757    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" event={"ID":"de6f1e1d-729d-40dc-bb32-b5421356a26e","Type":"ContainerStarted","Data":"45c31f93e4780a0eea72f2772af2853e25d08c0c3c0945077b4284ec1035bd49"}
Jul 27 17:23:40.627303 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:40.627203    2566 generic.go:356] "Generic (PLEG): container finished" podID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerID="47eb389ad123e1116885985a49c6a687bc22885245a3be7ce1b93b78757f40de" exitCode=0
Jul 27 17:23:40.627303 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:40.627275    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" event={"ID":"de6f1e1d-729d-40dc-bb32-b5421356a26e","Type":"ContainerDied","Data":"47eb389ad123e1116885985a49c6a687bc22885245a3be7ce1b93b78757f40de"}
Jul 27 17:23:41.640569 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:41.640534    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" event={"ID":"de6f1e1d-729d-40dc-bb32-b5421356a26e","Type":"ContainerStarted","Data":"41f7c8e9cd889b7b72a40dcc169d5fb2c807b3ca97db6d948fd4d4bdd1c534bc"}
Jul 27 17:23:41.665131 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:41.665062    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" podStartSLOduration=2.325601919 podStartE2EDuration="29.66504133s" totalImagesPullingTime="0s" totalInitContainerRuntime="27.339439411s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:23:41.662842436 +0000 UTC m=+1365.738106924" watchObservedRunningTime="2026-07-27 17:23:41.66504133 +0000 UTC m=+1365.740305821"
Jul 27 17:23:41.726118 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:41.726072    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:42.725243 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:42.725203    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:23:42.726887 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:42.726855    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" podUID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 27 17:23:50.264674 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:50.264641    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d"]
Jul 27 17:23:50.268305 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:50.268282    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d"
Jul 27 17:23:50.270939 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:50.270915    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 27 17:23:50.272239 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:50.272214    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-lctd5\""
Jul 27 17:23:50.272339 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:50.272265    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 27 17:23:50.272339 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:50.272312    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-hfh86\""
Jul 27 17:23:50.272464 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:50.272367    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 27 17:23:50.277959 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:50.277938    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d"]
Jul 27 17:23:50.333925 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:50.333886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f90d6364-a25f-4511-9659-f40717007882-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d\" (UID: \"f90d6364-a25f-4511-9659-f40717007882\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d"
Jul 27 17:23:50.334113 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:50.333960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2cpl\" (UniqueName: \"kubernetes.io/projected/f90d6364-a25f-4511-9659-f40717007882-kube-api-access-n2cpl\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d\" (UID: \"f90d6364-a25f-4511-9659-f40717007882\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d"
Jul 27 17:23:50.434649 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:50.434602    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f90d6364-a25f-4511-9659-f40717007882-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d\" (UID: \"f90d6364-a25f-4511-9659-f40717007882\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d"
Jul 27 17:23:50.434851 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:50.434681    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2cpl\" (UniqueName: \"kubernetes.io/projected/f90d6364-a25f-4511-9659-f40717007882-kube-api-access-n2cpl\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d\" (UID: \"f90d6364-a25f-4511-9659-f40717007882\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d"
Jul 27 17:23:50.437152 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:50.437117    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f90d6364-a25f-4511-9659-f40717007882-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d\" (UID: \"f90d6364-a25f-4511-9659-f40717007882\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d"
Jul 27 17:23:50.444594 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:50.444567    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2cpl\" (UniqueName: \"kubernetes.io/projected/f90d6364-a25f-4511-9659-f40717007882-kube-api-access-n2cpl\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d\" (UID: \"f90d6364-a25f-4511-9659-f40717007882\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d"
Jul 27 17:23:50.580109 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:50.580024    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d"
Jul 27 17:23:50.728653 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:50.728627    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d"]
Jul 27 17:23:50.730964 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:23:50.730933    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90d6364_a25f_4511_9659_f40717007882.slice/crio-924461cde64ff2e6b8f56c3e1b17d33f45ceafb1cff80bd9e28c040119a7b3ff WatchSource:0}: Error finding container 924461cde64ff2e6b8f56c3e1b17d33f45ceafb1cff80bd9e28c040119a7b3ff: Status 404 returned error can't find the container with id 924461cde64ff2e6b8f56c3e1b17d33f45ceafb1cff80bd9e28c040119a7b3ff
Jul 27 17:23:50.733698 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:50.733678    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:23:51.706952 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:51.706899    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d" event={"ID":"f90d6364-a25f-4511-9659-f40717007882","Type":"ContainerStarted","Data":"924461cde64ff2e6b8f56c3e1b17d33f45ceafb1cff80bd9e28c040119a7b3ff"}
Jul 27 17:23:52.726246 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:52.726204    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" podUID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 27 17:23:54.727836 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:54.727795    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d" event={"ID":"f90d6364-a25f-4511-9659-f40717007882","Type":"ContainerStarted","Data":"363f5ea2e455115aed7662737d75be45227210a4a5d6702b0b9f0d3f4116d871"}
Jul 27 17:23:54.728327 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:54.727864    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d"
Jul 27 17:23:54.745480 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:23:54.745436    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d" podStartSLOduration=1.656559143 podStartE2EDuration="4.745421909s" totalImagesPullingTime="3.088862766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:23:54.744148751 +0000 UTC m=+1378.819413239" watchObservedRunningTime="2026-07-27 17:23:54.745421909 +0000 UTC m=+1378.820686397"
Jul 27 17:24:02.726423 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:02.726379    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" podUID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 27 17:24:12.725888 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:12.725844    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" podUID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 27 17:24:22.726439 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:22.726385    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" podUID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 27 17:24:23.835416 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:23.835375    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d"]
Jul 27 17:24:23.839557 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:23.839309    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d" podUID="f90d6364-a25f-4511-9659-f40717007882" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:24:32.725994 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:32.725940    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" podUID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 27 17:24:33.838379 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:33.838334    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d" podUID="f90d6364-a25f-4511-9659-f40717007882" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:24:38.867576 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:38.867535    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d" podUID="f90d6364-a25f-4511-9659-f40717007882" containerName="main" containerID="cri-o://363f5ea2e455115aed7662737d75be45227210a4a5d6702b0b9f0d3f4116d871" gracePeriod=45
Jul 27 17:24:38.986635 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:38.986608    2566 generic.go:356] "Generic (PLEG): container finished" podID="f90d6364-a25f-4511-9659-f40717007882" containerID="363f5ea2e455115aed7662737d75be45227210a4a5d6702b0b9f0d3f4116d871" exitCode=0
Jul 27 17:24:38.986762 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:38.986691    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d" event={"ID":"f90d6364-a25f-4511-9659-f40717007882","Type":"ContainerDied","Data":"363f5ea2e455115aed7662737d75be45227210a4a5d6702b0b9f0d3f4116d871"}
Jul 27 17:24:39.119193 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:39.119092    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d"
Jul 27 17:24:39.193797 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:39.193752    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2cpl\" (UniqueName: \"kubernetes.io/projected/f90d6364-a25f-4511-9659-f40717007882-kube-api-access-n2cpl\") pod \"f90d6364-a25f-4511-9659-f40717007882\" (UID: \"f90d6364-a25f-4511-9659-f40717007882\") "
Jul 27 17:24:39.194010 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:39.193855    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f90d6364-a25f-4511-9659-f40717007882-tls-certs\") pod \"f90d6364-a25f-4511-9659-f40717007882\" (UID: \"f90d6364-a25f-4511-9659-f40717007882\") "
Jul 27 17:24:39.195974 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:39.195946    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f90d6364-a25f-4511-9659-f40717007882-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f90d6364-a25f-4511-9659-f40717007882" (UID: "f90d6364-a25f-4511-9659-f40717007882"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:24:39.196113 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:39.195949    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90d6364-a25f-4511-9659-f40717007882-kube-api-access-n2cpl" (OuterVolumeSpecName: "kube-api-access-n2cpl") pod "f90d6364-a25f-4511-9659-f40717007882" (UID: "f90d6364-a25f-4511-9659-f40717007882"). InnerVolumeSpecName "kube-api-access-n2cpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:24:39.295186 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:39.295145    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f90d6364-a25f-4511-9659-f40717007882-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:24:39.295186 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:39.295186    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2cpl\" (UniqueName: \"kubernetes.io/projected/f90d6364-a25f-4511-9659-f40717007882-kube-api-access-n2cpl\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:24:39.994318 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:39.994278    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d" event={"ID":"f90d6364-a25f-4511-9659-f40717007882","Type":"ContainerDied","Data":"924461cde64ff2e6b8f56c3e1b17d33f45ceafb1cff80bd9e28c040119a7b3ff"}
Jul 27 17:24:39.994816 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:39.994338    2566 scope.go:117] "RemoveContainer" containerID="363f5ea2e455115aed7662737d75be45227210a4a5d6702b0b9f0d3f4116d871"
Jul 27 17:24:39.994816 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:39.994289    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d"
Jul 27 17:24:40.016182 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:40.016143    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d"]
Jul 27 17:24:40.019920 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:40.019894    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678spg7d"]
Jul 27 17:24:40.404853 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:40.404775    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f90d6364-a25f-4511-9659-f40717007882" path="/var/lib/kubelet/pods/f90d6364-a25f-4511-9659-f40717007882/volumes"
Jul 27 17:24:42.725684 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:42.725634    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" podUID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 27 17:24:52.725562 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:24:52.725512    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" podUID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 27 17:25:02.726302 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:02.726255    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" podUID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 27 17:25:12.725797 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:12.725738    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" podUID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 27 17:25:22.725769 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:22.725721    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" podUID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 27 17:25:32.735333 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:32.735297    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:25:32.743333 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:32.743300    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:25:38.051666 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:38.051633    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"]
Jul 27 17:25:53.075662 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.075620    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" podUID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerName="main" containerID="cri-o://41f7c8e9cd889b7b72a40dcc169d5fb2c807b3ca97db6d948fd4d4bdd1c534bc" gracePeriod=45
Jul 27 17:25:53.736678 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.736636    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" podUID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 27 17:25:53.831884 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.831860    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:25:53.895572 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.895491    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jl5r9\" (UniqueName: \"kubernetes.io/projected/de6f1e1d-729d-40dc-bb32-b5421356a26e-kube-api-access-jl5r9\") pod \"de6f1e1d-729d-40dc-bb32-b5421356a26e\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") "
Jul 27 17:25:53.895572 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.895528    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-home\") pod \"de6f1e1d-729d-40dc-bb32-b5421356a26e\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") "
Jul 27 17:25:53.895746 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.895582    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-tmp-dir\") pod \"de6f1e1d-729d-40dc-bb32-b5421356a26e\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") "
Jul 27 17:25:53.895746 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.895734    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de6f1e1d-729d-40dc-bb32-b5421356a26e-tls-certs\") pod \"de6f1e1d-729d-40dc-bb32-b5421356a26e\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") "
Jul 27 17:25:53.895836 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.895822    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-kserve-provision-location\") pod \"de6f1e1d-729d-40dc-bb32-b5421356a26e\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") "
Jul 27 17:25:53.895894 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.895861    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-dshm\") pod \"de6f1e1d-729d-40dc-bb32-b5421356a26e\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") "
Jul 27 17:25:53.895950 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.895928    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-model-cache\") pod \"de6f1e1d-729d-40dc-bb32-b5421356a26e\" (UID: \"de6f1e1d-729d-40dc-bb32-b5421356a26e\") "
Jul 27 17:25:53.896355 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.896328    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-home" (OuterVolumeSpecName: "home") pod "de6f1e1d-729d-40dc-bb32-b5421356a26e" (UID: "de6f1e1d-729d-40dc-bb32-b5421356a26e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:25:53.896484 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.896351    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-model-cache" (OuterVolumeSpecName: "model-cache") pod "de6f1e1d-729d-40dc-bb32-b5421356a26e" (UID: "de6f1e1d-729d-40dc-bb32-b5421356a26e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:25:53.897989 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.897959    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de6f1e1d-729d-40dc-bb32-b5421356a26e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de6f1e1d-729d-40dc-bb32-b5421356a26e" (UID: "de6f1e1d-729d-40dc-bb32-b5421356a26e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:25:53.898157 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.898136    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de6f1e1d-729d-40dc-bb32-b5421356a26e-kube-api-access-jl5r9" (OuterVolumeSpecName: "kube-api-access-jl5r9") pod "de6f1e1d-729d-40dc-bb32-b5421356a26e" (UID: "de6f1e1d-729d-40dc-bb32-b5421356a26e"). InnerVolumeSpecName "kube-api-access-jl5r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:25:53.898372 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.898350    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-dshm" (OuterVolumeSpecName: "dshm") pod "de6f1e1d-729d-40dc-bb32-b5421356a26e" (UID: "de6f1e1d-729d-40dc-bb32-b5421356a26e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:25:53.908970 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.908947    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de6f1e1d-729d-40dc-bb32-b5421356a26e" (UID: "de6f1e1d-729d-40dc-bb32-b5421356a26e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:25:53.960671 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.960608    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de6f1e1d-729d-40dc-bb32-b5421356a26e" (UID: "de6f1e1d-729d-40dc-bb32-b5421356a26e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:25:53.996567 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.996540    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-model-cache\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:25:53.996567 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.996564    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl5r9\" (UniqueName: \"kubernetes.io/projected/de6f1e1d-729d-40dc-bb32-b5421356a26e-kube-api-access-jl5r9\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:25:53.996737 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.996574    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-home\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:25:53.996737 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.996583    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-tmp-dir\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:25:53.996737 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.996590    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de6f1e1d-729d-40dc-bb32-b5421356a26e-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:25:53.996737 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.996600    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-kserve-provision-location\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:25:53.996737 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:53.996608    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de6f1e1d-729d-40dc-bb32-b5421356a26e-dshm\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:25:54.445596 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:54.445562    2566 generic.go:356] "Generic (PLEG): container finished" podID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerID="41f7c8e9cd889b7b72a40dcc169d5fb2c807b3ca97db6d948fd4d4bdd1c534bc" exitCode=0
Jul 27 17:25:54.445992 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:54.445616    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"
Jul 27 17:25:54.445992 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:54.445621    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" event={"ID":"de6f1e1d-729d-40dc-bb32-b5421356a26e","Type":"ContainerDied","Data":"41f7c8e9cd889b7b72a40dcc169d5fb2c807b3ca97db6d948fd4d4bdd1c534bc"}
Jul 27 17:25:54.445992 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:54.445738    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm" event={"ID":"de6f1e1d-729d-40dc-bb32-b5421356a26e","Type":"ContainerDied","Data":"45c31f93e4780a0eea72f2772af2853e25d08c0c3c0945077b4284ec1035bd49"}
Jul 27 17:25:54.445992 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:54.445772    2566 scope.go:117] "RemoveContainer" containerID="41f7c8e9cd889b7b72a40dcc169d5fb2c807b3ca97db6d948fd4d4bdd1c534bc"
Jul 27 17:25:54.455025 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:54.455000    2566 scope.go:117] "RemoveContainer" containerID="47eb389ad123e1116885985a49c6a687bc22885245a3be7ce1b93b78757f40de"
Jul 27 17:25:54.463366 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:54.463348    2566 scope.go:117] "RemoveContainer" containerID="41f7c8e9cd889b7b72a40dcc169d5fb2c807b3ca97db6d948fd4d4bdd1c534bc"
Jul 27 17:25:54.463654 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:25:54.463632    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f7c8e9cd889b7b72a40dcc169d5fb2c807b3ca97db6d948fd4d4bdd1c534bc\": container with ID starting with 41f7c8e9cd889b7b72a40dcc169d5fb2c807b3ca97db6d948fd4d4bdd1c534bc not found: ID does not exist" containerID="41f7c8e9cd889b7b72a40dcc169d5fb2c807b3ca97db6d948fd4d4bdd1c534bc"
Jul 27 17:25:54.463707 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:54.463664    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f7c8e9cd889b7b72a40dcc169d5fb2c807b3ca97db6d948fd4d4bdd1c534bc"} err="failed to get container status \"41f7c8e9cd889b7b72a40dcc169d5fb2c807b3ca97db6d948fd4d4bdd1c534bc\": rpc error: code = NotFound desc = could not find container \"41f7c8e9cd889b7b72a40dcc169d5fb2c807b3ca97db6d948fd4d4bdd1c534bc\": container with ID starting with 41f7c8e9cd889b7b72a40dcc169d5fb2c807b3ca97db6d948fd4d4bdd1c534bc not found: ID does not exist"
Jul 27 17:25:54.463707 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:54.463681    2566 scope.go:117] "RemoveContainer" containerID="47eb389ad123e1116885985a49c6a687bc22885245a3be7ce1b93b78757f40de"
Jul 27 17:25:54.463936 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:25:54.463918    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47eb389ad123e1116885985a49c6a687bc22885245a3be7ce1b93b78757f40de\": container with ID starting with 47eb389ad123e1116885985a49c6a687bc22885245a3be7ce1b93b78757f40de not found: ID does not exist" containerID="47eb389ad123e1116885985a49c6a687bc22885245a3be7ce1b93b78757f40de"
Jul 27 17:25:54.463985 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:54.463944    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47eb389ad123e1116885985a49c6a687bc22885245a3be7ce1b93b78757f40de"} err="failed to get container status \"47eb389ad123e1116885985a49c6a687bc22885245a3be7ce1b93b78757f40de\": rpc error: code = NotFound desc = could not find container \"47eb389ad123e1116885985a49c6a687bc22885245a3be7ce1b93b78757f40de\": container with ID starting with 47eb389ad123e1116885985a49c6a687bc22885245a3be7ce1b93b78757f40de not found: ID does not exist"
Jul 27 17:25:54.470822 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:54.470799    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"]
Jul 27 17:25:54.474385 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:54.474362    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-755f6dd566-d8jsm"]
Jul 27 17:25:55.831650 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.831612    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"]
Jul 27 17:25:55.832033 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.832016    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerName="storage-initializer"
Jul 27 17:25:55.832078 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.832035    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerName="storage-initializer"
Jul 27 17:25:55.832078 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.832045    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerName="main"
Jul 27 17:25:55.832078 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.832054    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerName="main"
Jul 27 17:25:55.832078 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.832069    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f90d6364-a25f-4511-9659-f40717007882" containerName="main"
Jul 27 17:25:55.832078 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.832075    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f90d6364-a25f-4511-9659-f40717007882" containerName="main"
Jul 27 17:25:55.832253 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.832136    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f90d6364-a25f-4511-9659-f40717007882" containerName="main"
Jul 27 17:25:55.832253 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.832150    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="de6f1e1d-729d-40dc-bb32-b5421356a26e" containerName="main"
Jul 27 17:25:55.836793 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.836769    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:55.840321 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.840297    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-gn825\""
Jul 27 17:25:55.840321 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.840317    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 27 17:25:55.840487 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.840378    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 27 17:25:55.840487 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.840410    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 27 17:25:55.845808 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.845787    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"]
Jul 27 17:25:55.913413 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.913372    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:55.913600 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.913439    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8d6h\" (UniqueName: \"kubernetes.io/projected/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-kube-api-access-d8d6h\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:55.913600 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.913472    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:55.913600 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.913497    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:55.913600 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.913574    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:55.913747 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:55.913610    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:56.014251 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:56.014216    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:56.014251 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:56.014255    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:56.014497 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:56.014400    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:56.014497 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:56.014491    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8d6h\" (UniqueName: \"kubernetes.io/projected/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-kube-api-access-d8d6h\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:56.014614 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:56.014548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:56.014614 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:56.014599    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:56.014711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:56.014672    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:56.014711 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:56.014689    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:56.014913 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:56.014894    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:56.016681 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:56.016651    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:56.016885 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:56.016870    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:56.028916 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:56.028887    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8d6h\" (UniqueName: \"kubernetes.io/projected/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-kube-api-access-d8d6h\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:56.148402 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:56.148322    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:56.280187 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:56.280153    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"]
Jul 27 17:25:56.281389 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:25:56.281356    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e097dc7_4e31_4c5d_b636_e61c0f3ee647.slice/crio-3f30a65ddd3c1b8baf49ce8ec9192ecc5927e3ececa62a5f51fabbee2f37fb2d WatchSource:0}: Error finding container 3f30a65ddd3c1b8baf49ce8ec9192ecc5927e3ececa62a5f51fabbee2f37fb2d: Status 404 returned error can't find the container with id 3f30a65ddd3c1b8baf49ce8ec9192ecc5927e3ececa62a5f51fabbee2f37fb2d
Jul 27 17:25:56.403768 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:56.403699    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de6f1e1d-729d-40dc-bb32-b5421356a26e" path="/var/lib/kubelet/pods/de6f1e1d-729d-40dc-bb32-b5421356a26e/volumes"
Jul 27 17:25:56.458988 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:56.458948    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l" event={"ID":"0e097dc7-4e31-4c5d-b636-e61c0f3ee647","Type":"ContainerStarted","Data":"227a355b6cd4a778f919c693344d63da9a6844c2e4bc0bad54fac315eb1bc9f1"}
Jul 27 17:25:56.459144 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:56.459013    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l" event={"ID":"0e097dc7-4e31-4c5d-b636-e61c0f3ee647","Type":"ContainerStarted","Data":"3f30a65ddd3c1b8baf49ce8ec9192ecc5927e3ececa62a5f51fabbee2f37fb2d"}
Jul 27 17:25:56.478845 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:56.478794    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l" podStartSLOduration=1.478777108 podStartE2EDuration="1.478777108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:25:56.476459741 +0000 UTC m=+1500.551724229" watchObservedRunningTime="2026-07-27 17:25:56.478777108 +0000 UTC m=+1500.554041595"
Jul 27 17:25:57.148832 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:57.148795    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:25:57.428573 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:57.428489    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:25:57.432500 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:25:57.432480    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:26:06.149035 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:26:06.148999    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:26:06.161544 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:26:06.161515    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:26:06.525656 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:26:06.525621    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:27:35.506992 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:35.506953    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"]
Jul 27 17:27:35.516447 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:27:35.516410    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:27:35.516447 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:27:35.516447 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:27:35.516447 ip-10-0-143-86 kubenswrapper[2566]:  > containerID="227a355b6cd4a778f919c693344d63da9a6844c2e4bc0bad54fac315eb1bc9f1" cmd=["/bin/sleep","15"]
Jul 27 17:27:35.516765 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:35.516468    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l" message="" err=<
Jul 27 17:27:35.516765 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:27:35.516765 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:27:35.516765 ip-10-0-143-86 kubenswrapper[2566]:  >
Jul 27 17:27:35.516765 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:27:35.516484    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:27:35.516765 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:27:35.516765 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:27:35.516765 ip-10-0-143-86 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l" podUID="0e097dc7-4e31-4c5d-b636-e61c0f3ee647" containerName="main" containerID="cri-o://227a355b6cd4a778f919c693344d63da9a6844c2e4bc0bad54fac315eb1bc9f1"
Jul 27 17:27:35.516765 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:35.516539    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l" podUID="0e097dc7-4e31-4c5d-b636-e61c0f3ee647" containerName="main" containerID="cri-o://227a355b6cd4a778f919c693344d63da9a6844c2e4bc0bad54fac315eb1bc9f1" gracePeriod=60
Jul 27 17:27:36.006807 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.006772    2566 generic.go:356] "Generic (PLEG): container finished" podID="0e097dc7-4e31-4c5d-b636-e61c0f3ee647" containerID="227a355b6cd4a778f919c693344d63da9a6844c2e4bc0bad54fac315eb1bc9f1" exitCode=0
Jul 27 17:27:36.006988 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.006811    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l" event={"ID":"0e097dc7-4e31-4c5d-b636-e61c0f3ee647","Type":"ContainerDied","Data":"227a355b6cd4a778f919c693344d63da9a6844c2e4bc0bad54fac315eb1bc9f1"}
Jul 27 17:27:36.275681 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.275655    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:27:36.335528 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.335497    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8d6h\" (UniqueName: \"kubernetes.io/projected/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-kube-api-access-d8d6h\") pod \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") "
Jul 27 17:27:36.335713 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.335545    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-model-cache\") pod \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") "
Jul 27 17:27:36.335713 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.335577    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-dshm\") pod \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") "
Jul 27 17:27:36.335713 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.335615    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-tls-certs\") pod \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") "
Jul 27 17:27:36.335886 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.335720    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-home\") pod \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") "
Jul 27 17:27:36.335886 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.335747    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-tmp-dir\") pod \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\" (UID: \"0e097dc7-4e31-4c5d-b636-e61c0f3ee647\") "
Jul 27 17:27:36.335886 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.335825    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e097dc7-4e31-4c5d-b636-e61c0f3ee647" (UID: "0e097dc7-4e31-4c5d-b636-e61c0f3ee647"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:27:36.336040 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.336000    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-home" (OuterVolumeSpecName: "home") pod "0e097dc7-4e31-4c5d-b636-e61c0f3ee647" (UID: "0e097dc7-4e31-4c5d-b636-e61c0f3ee647"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:27:36.336097 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.336016    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-model-cache\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:27:36.336097 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.336052    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e097dc7-4e31-4c5d-b636-e61c0f3ee647" (UID: "0e097dc7-4e31-4c5d-b636-e61c0f3ee647"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:27:36.337780 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.337751    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-dshm" (OuterVolumeSpecName: "dshm") pod "0e097dc7-4e31-4c5d-b636-e61c0f3ee647" (UID: "0e097dc7-4e31-4c5d-b636-e61c0f3ee647"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:27:36.337888 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.337804    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-kube-api-access-d8d6h" (OuterVolumeSpecName: "kube-api-access-d8d6h") pod "0e097dc7-4e31-4c5d-b636-e61c0f3ee647" (UID: "0e097dc7-4e31-4c5d-b636-e61c0f3ee647"). InnerVolumeSpecName "kube-api-access-d8d6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:27:36.337888 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.337825    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e097dc7-4e31-4c5d-b636-e61c0f3ee647" (UID: "0e097dc7-4e31-4c5d-b636-e61c0f3ee647"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:27:36.437315 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.437280    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-dshm\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:27:36.437315 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.437311    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:27:36.437315 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.437323    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-home\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:27:36.437550 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.437330    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-tmp-dir\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:27:36.437550 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:36.437340    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8d6h\" (UniqueName: \"kubernetes.io/projected/0e097dc7-4e31-4c5d-b636-e61c0f3ee647-kube-api-access-d8d6h\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:27:37.012971 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:37.012943    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"
Jul 27 17:27:37.013511 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:37.012987    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l" event={"ID":"0e097dc7-4e31-4c5d-b636-e61c0f3ee647","Type":"ContainerDied","Data":"3f30a65ddd3c1b8baf49ce8ec9192ecc5927e3ececa62a5f51fabbee2f37fb2d"}
Jul 27 17:27:37.013511 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:37.013060    2566 scope.go:117] "RemoveContainer" containerID="227a355b6cd4a778f919c693344d63da9a6844c2e4bc0bad54fac315eb1bc9f1"
Jul 27 17:27:37.030547 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:37.030515    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"]
Jul 27 17:27:37.035853 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:37.035822    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-m969l"]
Jul 27 17:27:38.402842 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:27:38.402804    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e097dc7-4e31-4c5d-b636-e61c0f3ee647" path="/var/lib/kubelet/pods/0e097dc7-4e31-4c5d-b636-e61c0f3ee647/volumes"
Jul 27 17:28:01.036547 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.036512    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"]
Jul 27 17:28:01.036948 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.036897    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e097dc7-4e31-4c5d-b636-e61c0f3ee647" containerName="main"
Jul 27 17:28:01.036948 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.036910    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e097dc7-4e31-4c5d-b636-e61c0f3ee647" containerName="main"
Jul 27 17:28:01.037026 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.036984    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e097dc7-4e31-4c5d-b636-e61c0f3ee647" containerName="main"
Jul 27 17:28:01.040096 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.040071    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.044263 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.044229    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 27 17:28:01.044394 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.044229    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 27 17:28:01.044394 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.044229    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-95l2n\""
Jul 27 17:28:01.044518 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.044229    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 27 17:28:01.051895 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.051875    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"]
Jul 27 17:28:01.158371 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.158338    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.158371 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.158371    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.158578 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.158403    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c569972-6057-4535-8224-3cfba0d664bd-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.158578 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.158422    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.158578 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.158445    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.158578 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.158470    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4nng\" (UniqueName: \"kubernetes.io/projected/3c569972-6057-4535-8224-3cfba0d664bd-kube-api-access-s4nng\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.259144 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.259105    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c569972-6057-4535-8224-3cfba0d664bd-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.259144 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.259149    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.259391 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.259194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.259391 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.259223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4nng\" (UniqueName: \"kubernetes.io/projected/3c569972-6057-4535-8224-3cfba0d664bd-kube-api-access-s4nng\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.259391 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.259295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.259391 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.259320    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.259775 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.259748    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.259775 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.259769    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.259943 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.259804    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.261770 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.261737    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.262346 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.262325    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c569972-6057-4535-8224-3cfba0d664bd-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.273657 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.273629    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4nng\" (UniqueName: \"kubernetes.io/projected/3c569972-6057-4535-8224-3cfba0d664bd-kube-api-access-s4nng\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.351719 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.351632    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:01.489625 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:01.489586    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"]
Jul 27 17:28:01.491643 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:28:01.491614    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c569972_6057_4535_8224_3cfba0d664bd.slice/crio-147293bdc0652fd17f9b38fe8c52926100e95dc45e8d704caa494fa652f26162 WatchSource:0}: Error finding container 147293bdc0652fd17f9b38fe8c52926100e95dc45e8d704caa494fa652f26162: Status 404 returned error can't find the container with id 147293bdc0652fd17f9b38fe8c52926100e95dc45e8d704caa494fa652f26162
Jul 27 17:28:02.149138 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:02.149096    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb" event={"ID":"3c569972-6057-4535-8224-3cfba0d664bd","Type":"ContainerStarted","Data":"6eb0eb13e8d91af11b57f4bae9e967ab7084cf42ccf51580f27ef58dfefa477d"}
Jul 27 17:28:02.149622 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:02.149161    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb" event={"ID":"3c569972-6057-4535-8224-3cfba0d664bd","Type":"ContainerStarted","Data":"147293bdc0652fd17f9b38fe8c52926100e95dc45e8d704caa494fa652f26162"}
Jul 27 17:28:02.170223 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:02.170153    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb" podStartSLOduration=1.170136891 podStartE2EDuration="1.170136891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:02.167991858 +0000 UTC m=+1626.243256344" watchObservedRunningTime="2026-07-27 17:28:02.170136891 +0000 UTC m=+1626.245401377"
Jul 27 17:28:02.352500 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:02.352466    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:11.352704 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:11.352663    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:11.365056 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:11.365021    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:12.212895 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:12.212858    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:34.840258 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:34.840219    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"]
Jul 27 17:28:34.849194 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:28:34.849146    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:28:34.849194 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:28:34.849194 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:28:34.849194 ip-10-0-143-86 kubenswrapper[2566]:  > containerID="6eb0eb13e8d91af11b57f4bae9e967ab7084cf42ccf51580f27ef58dfefa477d" cmd=["/bin/sleep","15"]
Jul 27 17:28:34.849514 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:34.849210    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb" message="" err=<
Jul 27 17:28:34.849514 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:28:34.849514 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:28:34.849514 ip-10-0-143-86 kubenswrapper[2566]:  >
Jul 27 17:28:34.849514 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:28:34.849223    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:28:34.849514 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:28:34.849514 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:28:34.849514 ip-10-0-143-86 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb" podUID="3c569972-6057-4535-8224-3cfba0d664bd" containerName="main" containerID="cri-o://6eb0eb13e8d91af11b57f4bae9e967ab7084cf42ccf51580f27ef58dfefa477d"
Jul 27 17:28:34.849514 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:34.849266    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb" podUID="3c569972-6057-4535-8224-3cfba0d664bd" containerName="main" containerID="cri-o://6eb0eb13e8d91af11b57f4bae9e967ab7084cf42ccf51580f27ef58dfefa477d" gracePeriod=60
Jul 27 17:28:35.211300 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.211276    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:35.338466 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.338426    2566 generic.go:356] "Generic (PLEG): container finished" podID="3c569972-6057-4535-8224-3cfba0d664bd" containerID="6eb0eb13e8d91af11b57f4bae9e967ab7084cf42ccf51580f27ef58dfefa477d" exitCode=0
Jul 27 17:28:35.338679 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.338476    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"
Jul 27 17:28:35.338679 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.338517    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb" event={"ID":"3c569972-6057-4535-8224-3cfba0d664bd","Type":"ContainerDied","Data":"6eb0eb13e8d91af11b57f4bae9e967ab7084cf42ccf51580f27ef58dfefa477d"}
Jul 27 17:28:35.338679 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.338588    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb" event={"ID":"3c569972-6057-4535-8224-3cfba0d664bd","Type":"ContainerDied","Data":"147293bdc0652fd17f9b38fe8c52926100e95dc45e8d704caa494fa652f26162"}
Jul 27 17:28:35.338679 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.338622    2566 scope.go:117] "RemoveContainer" containerID="6eb0eb13e8d91af11b57f4bae9e967ab7084cf42ccf51580f27ef58dfefa477d"
Jul 27 17:28:35.350048 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.350020    2566 scope.go:117] "RemoveContainer" containerID="6eb0eb13e8d91af11b57f4bae9e967ab7084cf42ccf51580f27ef58dfefa477d"
Jul 27 17:28:35.350361 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:28:35.350339    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb0eb13e8d91af11b57f4bae9e967ab7084cf42ccf51580f27ef58dfefa477d\": container with ID starting with 6eb0eb13e8d91af11b57f4bae9e967ab7084cf42ccf51580f27ef58dfefa477d not found: ID does not exist" containerID="6eb0eb13e8d91af11b57f4bae9e967ab7084cf42ccf51580f27ef58dfefa477d"
Jul 27 17:28:35.350464 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.350370    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb0eb13e8d91af11b57f4bae9e967ab7084cf42ccf51580f27ef58dfefa477d"} err="failed to get container status \"6eb0eb13e8d91af11b57f4bae9e967ab7084cf42ccf51580f27ef58dfefa477d\": rpc error: code = NotFound desc = could not find container \"6eb0eb13e8d91af11b57f4bae9e967ab7084cf42ccf51580f27ef58dfefa477d\": container with ID starting with 6eb0eb13e8d91af11b57f4bae9e967ab7084cf42ccf51580f27ef58dfefa477d not found: ID does not exist"
Jul 27 17:28:35.372886 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.372801    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c569972-6057-4535-8224-3cfba0d664bd-tls-certs\") pod \"3c569972-6057-4535-8224-3cfba0d664bd\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") "
Jul 27 17:28:35.372886 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.372839    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-dshm\") pod \"3c569972-6057-4535-8224-3cfba0d664bd\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") "
Jul 27 17:28:35.372886 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.372881    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-home\") pod \"3c569972-6057-4535-8224-3cfba0d664bd\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") "
Jul 27 17:28:35.373152 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.372948    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-tmp-dir\") pod \"3c569972-6057-4535-8224-3cfba0d664bd\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") "
Jul 27 17:28:35.373152 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.372975    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4nng\" (UniqueName: \"kubernetes.io/projected/3c569972-6057-4535-8224-3cfba0d664bd-kube-api-access-s4nng\") pod \"3c569972-6057-4535-8224-3cfba0d664bd\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") "
Jul 27 17:28:35.373152 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.373036    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-model-cache\") pod \"3c569972-6057-4535-8224-3cfba0d664bd\" (UID: \"3c569972-6057-4535-8224-3cfba0d664bd\") "
Jul 27 17:28:35.373479 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.373352    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c569972-6057-4535-8224-3cfba0d664bd" (UID: "3c569972-6057-4535-8224-3cfba0d664bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:28:35.373479 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.373446    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-home" (OuterVolumeSpecName: "home") pod "3c569972-6057-4535-8224-3cfba0d664bd" (UID: "3c569972-6057-4535-8224-3cfba0d664bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:28:35.373773 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.373582    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c569972-6057-4535-8224-3cfba0d664bd" (UID: "3c569972-6057-4535-8224-3cfba0d664bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:28:35.375439 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.375402    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c569972-6057-4535-8224-3cfba0d664bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c569972-6057-4535-8224-3cfba0d664bd" (UID: "3c569972-6057-4535-8224-3cfba0d664bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:28:35.375671 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.375639    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c569972-6057-4535-8224-3cfba0d664bd-kube-api-access-s4nng" (OuterVolumeSpecName: "kube-api-access-s4nng") pod "3c569972-6057-4535-8224-3cfba0d664bd" (UID: "3c569972-6057-4535-8224-3cfba0d664bd"). InnerVolumeSpecName "kube-api-access-s4nng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:28:35.375783 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.375738    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-dshm" (OuterVolumeSpecName: "dshm") pod "3c569972-6057-4535-8224-3cfba0d664bd" (UID: "3c569972-6057-4535-8224-3cfba0d664bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:28:35.474378 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.474335    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-home\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:28:35.474378 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.474370    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-tmp-dir\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:28:35.474378 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.474379    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4nng\" (UniqueName: \"kubernetes.io/projected/3c569972-6057-4535-8224-3cfba0d664bd-kube-api-access-s4nng\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:28:35.474378 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.474389    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-model-cache\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:28:35.474662 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.474398    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c569972-6057-4535-8224-3cfba0d664bd-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:28:35.474662 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.474406    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c569972-6057-4535-8224-3cfba0d664bd-dshm\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:28:35.662123 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.662092    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"]
Jul 27 17:28:35.666548 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:35.666522    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb"]
Jul 27 17:28:36.203265 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:36.203222    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58fx7tkb" podUID="3c569972-6057-4535-8224-3cfba0d664bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 27 17:28:36.403050 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:28:36.403011    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c569972-6057-4535-8224-3cfba0d664bd" path="/var/lib/kubelet/pods/3c569972-6057-4535-8224-3cfba0d664bd/volumes"
Jul 27 17:29:06.712479 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.712444    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"]
Jul 27 17:29:06.712937 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.712820    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c569972-6057-4535-8224-3cfba0d664bd" containerName="main"
Jul 27 17:29:06.712937 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.712831    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c569972-6057-4535-8224-3cfba0d664bd" containerName="main"
Jul 27 17:29:06.712937 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.712884    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c569972-6057-4535-8224-3cfba0d664bd" containerName="main"
Jul 27 17:29:06.716024 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.716001    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.722646 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.722622    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Jul 27 17:29:06.722895 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.722875    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 27 17:29:06.723863 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.723847    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-945qz\""
Jul 27 17:29:06.728494 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.728478    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 27 17:29:06.728571 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.728554    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 27 17:29:06.754356 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.754314    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"]
Jul 27 17:29:06.862072 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.862036    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-model-cache\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.862314 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.862089    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-dshm\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.862314 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.862221    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.862314 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.862266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4j98\" (UniqueName: \"kubernetes.io/projected/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-kube-api-access-z4j98\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.862314 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.862304    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.862516 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.862336    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-home\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.862516 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.862379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.963088 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.963002    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4j98\" (UniqueName: \"kubernetes.io/projected/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-kube-api-access-z4j98\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.963088 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.963046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.963088 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.963073    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-home\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.963394 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.963119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.963394 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.963188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-model-cache\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.963394 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.963227    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-dshm\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.963394 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.963272    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.963609 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.963503    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.963609 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.963548    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-home\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.963609 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.963571    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.963720 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.963616    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-model-cache\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.965457 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.965433    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-dshm\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.965767 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.965750    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:06.971052 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:06.971031    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4j98\" (UniqueName: \"kubernetes.io/projected/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-kube-api-access-z4j98\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:07.013229 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:07.013191    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"]
Jul 27 17:29:07.017406 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:07.017381    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"
Jul 27 17:29:07.026385 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:07.026135    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:07.028334 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:07.028294    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"]
Jul 27 17:29:07.064131 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:07.064080    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22a27948-bc69-4336-b004-cf8c1455ecd0-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx\" (UID: \"22a27948-bc69-4336-b004-cf8c1455ecd0\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"
Jul 27 17:29:07.064349 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:07.064140    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a27948-bc69-4336-b004-cf8c1455ecd0-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx\" (UID: \"22a27948-bc69-4336-b004-cf8c1455ecd0\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"
Jul 27 17:29:07.064349 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:07.064267    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a27948-bc69-4336-b004-cf8c1455ecd0-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx\" (UID: \"22a27948-bc69-4336-b004-cf8c1455ecd0\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"
Jul 27 17:29:07.165073 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:07.165036    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a27948-bc69-4336-b004-cf8c1455ecd0-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx\" (UID: \"22a27948-bc69-4336-b004-cf8c1455ecd0\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"
Jul 27 17:29:07.165264 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:07.165108    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a27948-bc69-4336-b004-cf8c1455ecd0-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx\" (UID: \"22a27948-bc69-4336-b004-cf8c1455ecd0\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"
Jul 27 17:29:07.165264 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:07.165192    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22a27948-bc69-4336-b004-cf8c1455ecd0-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx\" (UID: \"22a27948-bc69-4336-b004-cf8c1455ecd0\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"
Jul 27 17:29:07.165526 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:07.165500    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a27948-bc69-4336-b004-cf8c1455ecd0-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx\" (UID: \"22a27948-bc69-4336-b004-cf8c1455ecd0\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"
Jul 27 17:29:07.165596 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:07.165513    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22a27948-bc69-4336-b004-cf8c1455ecd0-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx\" (UID: \"22a27948-bc69-4336-b004-cf8c1455ecd0\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"
Jul 27 17:29:07.169111 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:07.168984    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a27948-bc69-4336-b004-cf8c1455ecd0-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx\" (UID: \"22a27948-bc69-4336-b004-cf8c1455ecd0\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"
Jul 27 17:29:07.170096 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:07.170075    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"]
Jul 27 17:29:07.172067 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:29:07.172041    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6131eb4_cf32_4b03_ba85_9fb59c5e5b0a.slice/crio-89e3744a354f1fa8ca303208f85ce965ca64d6618c57ea298fb64d53b3f34e34 WatchSource:0}: Error finding container 89e3744a354f1fa8ca303208f85ce965ca64d6618c57ea298fb64d53b3f34e34: Status 404 returned error can't find the container with id 89e3744a354f1fa8ca303208f85ce965ca64d6618c57ea298fb64d53b3f34e34
Jul 27 17:29:07.333605 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:07.333573    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"
Jul 27 17:29:07.474607 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:07.474583    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"]
Jul 27 17:29:07.476824 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:29:07.476797    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a27948_bc69_4336_b004_cf8c1455ecd0.slice/crio-a393e2ba9ea0286b43bf7122a84be3618336749ed8d41cba4e5edfe4952cd352 WatchSource:0}: Error finding container a393e2ba9ea0286b43bf7122a84be3618336749ed8d41cba4e5edfe4952cd352: Status 404 returned error can't find the container with id a393e2ba9ea0286b43bf7122a84be3618336749ed8d41cba4e5edfe4952cd352
Jul 27 17:29:07.520445 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:07.520403    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx" event={"ID":"22a27948-bc69-4336-b004-cf8c1455ecd0","Type":"ContainerStarted","Data":"a393e2ba9ea0286b43bf7122a84be3618336749ed8d41cba4e5edfe4952cd352"}
Jul 27 17:29:07.521985 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:07.521960    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7" event={"ID":"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a","Type":"ContainerStarted","Data":"0a0364cbe1b5ecc52b191a90323f605933cb62d6a2ea2faa32a29abf1d437c3b"}
Jul 27 17:29:07.522096 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:07.522013    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7" event={"ID":"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a","Type":"ContainerStarted","Data":"89e3744a354f1fa8ca303208f85ce965ca64d6618c57ea298fb64d53b3f34e34"}
Jul 27 17:29:08.537715 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:08.537671    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx" event={"ID":"22a27948-bc69-4336-b004-cf8c1455ecd0","Type":"ContainerStarted","Data":"ad9a9a0c68a1e95e1fdeaaa84a69e79e1e964db9dc65137eb2a98077552ac05a"}
Jul 27 17:29:14.576629 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:14.576593    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a" containerID="0a0364cbe1b5ecc52b191a90323f605933cb62d6a2ea2faa32a29abf1d437c3b" exitCode=0
Jul 27 17:29:14.577037 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:14.576684    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7" event={"ID":"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a","Type":"ContainerDied","Data":"0a0364cbe1b5ecc52b191a90323f605933cb62d6a2ea2faa32a29abf1d437c3b"}
Jul 27 17:29:15.584056 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:15.584022    2566 generic.go:356] "Generic (PLEG): container finished" podID="22a27948-bc69-4336-b004-cf8c1455ecd0" containerID="ad9a9a0c68a1e95e1fdeaaa84a69e79e1e964db9dc65137eb2a98077552ac05a" exitCode=0
Jul 27 17:29:15.584542 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:15.584104    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx" event={"ID":"22a27948-bc69-4336-b004-cf8c1455ecd0","Type":"ContainerDied","Data":"ad9a9a0c68a1e95e1fdeaaa84a69e79e1e964db9dc65137eb2a98077552ac05a"}
Jul 27 17:29:15.585324 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:15.585313    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:29:15.586052 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:15.586023    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7" event={"ID":"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a","Type":"ContainerStarted","Data":"71d4d70dfb2b6989b11c29610e05ee3702ec72596a2feecacf8354ace34f0191"}
Jul 27 17:29:15.623586 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:15.623528    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7" podStartSLOduration=2.987116256 podStartE2EDuration="9.623514436s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.63639818s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:15.622121228 +0000 UTC m=+1699.697385715" watchObservedRunningTime="2026-07-27 17:29:15.623514436 +0000 UTC m=+1699.698778967"
Jul 27 17:29:16.028128 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:16.028087    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:17.027849 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:17.027814    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:17.046231 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:17.046204    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:17.615562 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:17.615494    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:29:55.846264 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:55.846223    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx" event={"ID":"22a27948-bc69-4336-b004-cf8c1455ecd0","Type":"ContainerStarted","Data":"55bb3128b52c39c2b0a3059694d3e1bb6015dd2875f61dab2bbc926dcf4ae77e"}
Jul 27 17:29:55.846692 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:55.846472    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"
Jul 27 17:29:55.864819 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:29:55.864765    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx" podStartSLOduration=2.9297517170000003 podStartE2EDuration="49.86475086s" totalImagesPullingTime="39.759548275s" totalInitContainerRuntime="7.175450868s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:55.86238297 +0000 UTC m=+1739.937647459" watchObservedRunningTime="2026-07-27 17:29:55.86475086 +0000 UTC m=+1739.940015346"
Jul 27 17:30:26.864592 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:26.864556    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"
Jul 27 17:30:28.107576 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:28.107542    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx_22a27948-bc69-4336-b004-cf8c1455ecd0/main/0.log"
Jul 27 17:30:28.571089 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:28.571058    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"]
Jul 27 17:30:28.575523 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:28.575495    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"]
Jul 27 17:30:28.575725 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:28.575701    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx" podUID="22a27948-bc69-4336-b004-cf8c1455ecd0" containerName="main" containerID="cri-o://55bb3128b52c39c2b0a3059694d3e1bb6015dd2875f61dab2bbc926dcf4ae77e" gracePeriod=30
Jul 27 17:30:28.582481 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:30:28.582446    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:30:28.582481 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:30:28.582481 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:30:28.582481 ip-10-0-143-86 kubenswrapper[2566]:  > containerID="71d4d70dfb2b6989b11c29610e05ee3702ec72596a2feecacf8354ace34f0191" cmd=["/bin/sleep","15"]
Jul 27 17:30:28.582748 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:28.582506    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7" message="" err=<
Jul 27 17:30:28.582748 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:30:28.582748 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:30:28.582748 ip-10-0-143-86 kubenswrapper[2566]:  >
Jul 27 17:30:28.582748 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:30:28.582524    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:30:28.582748 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:30:28.582748 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:30:28.582748 ip-10-0-143-86 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7" podUID="c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a" containerName="main" containerID="cri-o://71d4d70dfb2b6989b11c29610e05ee3702ec72596a2feecacf8354ace34f0191"
Jul 27 17:30:28.582748 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:28.582595    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7" podUID="c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a" containerName="main" containerID="cri-o://71d4d70dfb2b6989b11c29610e05ee3702ec72596a2feecacf8354ace34f0191" gracePeriod=60
Jul 27 17:30:28.601077 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:28.601048    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7" podUID="c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Jul 27 17:30:28.963051 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:28.963021    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:30:29.044689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.044652    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a" containerID="71d4d70dfb2b6989b11c29610e05ee3702ec72596a2feecacf8354ace34f0191" exitCode=0
Jul 27 17:30:29.044883 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.044713    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"
Jul 27 17:30:29.044883 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.044740    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7" event={"ID":"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a","Type":"ContainerDied","Data":"71d4d70dfb2b6989b11c29610e05ee3702ec72596a2feecacf8354ace34f0191"}
Jul 27 17:30:29.044883 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.044806    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7" event={"ID":"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a","Type":"ContainerDied","Data":"89e3744a354f1fa8ca303208f85ce965ca64d6618c57ea298fb64d53b3f34e34"}
Jul 27 17:30:29.044883 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.044875    2566 scope.go:117] "RemoveContainer" containerID="71d4d70dfb2b6989b11c29610e05ee3702ec72596a2feecacf8354ace34f0191"
Jul 27 17:30:29.056059 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.056033    2566 scope.go:117] "RemoveContainer" containerID="0a0364cbe1b5ecc52b191a90323f605933cb62d6a2ea2faa32a29abf1d437c3b"
Jul 27 17:30:29.057431 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.057152    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-tmp-dir\") pod \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") "
Jul 27 17:30:29.057431 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.057217    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-kserve-provision-location\") pod \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") "
Jul 27 17:30:29.057431 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.057301    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-dshm\") pod \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") "
Jul 27 17:30:29.057431 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.057332    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4j98\" (UniqueName: \"kubernetes.io/projected/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-kube-api-access-z4j98\") pod \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") "
Jul 27 17:30:29.057431 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.057355    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-home\") pod \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") "
Jul 27 17:30:29.057431 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.057381    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-model-cache\") pod \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") "
Jul 27 17:30:29.057431 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.057418    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-tls-certs\") pod \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\" (UID: \"c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a\") "
Jul 27 17:30:29.058490 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.058430    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a" (UID: "c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:30:29.058490 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.058443    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-home" (OuterVolumeSpecName: "home") pod "c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a" (UID: "c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:30:29.058848 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.058827    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a" (UID: "c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:30:29.060902 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.060872    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-dshm" (OuterVolumeSpecName: "dshm") pod "c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a" (UID: "c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:30:29.061082 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.061057    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a" (UID: "c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:30:29.062782 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.062748    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-kube-api-access-z4j98" (OuterVolumeSpecName: "kube-api-access-z4j98") pod "c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a" (UID: "c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a"). InnerVolumeSpecName "kube-api-access-z4j98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:30:29.126126 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.126098    2566 scope.go:117] "RemoveContainer" containerID="71d4d70dfb2b6989b11c29610e05ee3702ec72596a2feecacf8354ace34f0191"
Jul 27 17:30:29.126533 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:30:29.126501    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d4d70dfb2b6989b11c29610e05ee3702ec72596a2feecacf8354ace34f0191\": container with ID starting with 71d4d70dfb2b6989b11c29610e05ee3702ec72596a2feecacf8354ace34f0191 not found: ID does not exist" containerID="71d4d70dfb2b6989b11c29610e05ee3702ec72596a2feecacf8354ace34f0191"
Jul 27 17:30:29.126598 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.126547    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d4d70dfb2b6989b11c29610e05ee3702ec72596a2feecacf8354ace34f0191"} err="failed to get container status \"71d4d70dfb2b6989b11c29610e05ee3702ec72596a2feecacf8354ace34f0191\": rpc error: code = NotFound desc = could not find container \"71d4d70dfb2b6989b11c29610e05ee3702ec72596a2feecacf8354ace34f0191\": container with ID starting with 71d4d70dfb2b6989b11c29610e05ee3702ec72596a2feecacf8354ace34f0191 not found: ID does not exist"
Jul 27 17:30:29.126598 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.126578    2566 scope.go:117] "RemoveContainer" containerID="0a0364cbe1b5ecc52b191a90323f605933cb62d6a2ea2faa32a29abf1d437c3b"
Jul 27 17:30:29.126956 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:30:29.126883    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a0364cbe1b5ecc52b191a90323f605933cb62d6a2ea2faa32a29abf1d437c3b\": container with ID starting with 0a0364cbe1b5ecc52b191a90323f605933cb62d6a2ea2faa32a29abf1d437c3b not found: ID does not exist" containerID="0a0364cbe1b5ecc52b191a90323f605933cb62d6a2ea2faa32a29abf1d437c3b"
Jul 27 17:30:29.126956 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.126915    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a0364cbe1b5ecc52b191a90323f605933cb62d6a2ea2faa32a29abf1d437c3b"} err="failed to get container status \"0a0364cbe1b5ecc52b191a90323f605933cb62d6a2ea2faa32a29abf1d437c3b\": rpc error: code = NotFound desc = could not find container \"0a0364cbe1b5ecc52b191a90323f605933cb62d6a2ea2faa32a29abf1d437c3b\": container with ID starting with 0a0364cbe1b5ecc52b191a90323f605933cb62d6a2ea2faa32a29abf1d437c3b not found: ID does not exist"
Jul 27 17:30:29.139771 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.139623    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a" (UID: "c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:30:29.158727 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.158641    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-tmp-dir\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:30:29.158727 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.158683    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-kserve-provision-location\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:30:29.158727 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.158698    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-dshm\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:30:29.158727 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.158711    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4j98\" (UniqueName: \"kubernetes.io/projected/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-kube-api-access-z4j98\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:30:29.158727 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.158723    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-home\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:30:29.158727 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.158736    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-model-cache\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:30:29.159040 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.158749    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:30:29.302079 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.302045    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"
Jul 27 17:30:29.359809 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.359781    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a27948-bc69-4336-b004-cf8c1455ecd0-tls-certs\") pod \"22a27948-bc69-4336-b004-cf8c1455ecd0\" (UID: \"22a27948-bc69-4336-b004-cf8c1455ecd0\") "
Jul 27 17:30:29.359963 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.359860    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22a27948-bc69-4336-b004-cf8c1455ecd0-tmp-dir\") pod \"22a27948-bc69-4336-b004-cf8c1455ecd0\" (UID: \"22a27948-bc69-4336-b004-cf8c1455ecd0\") "
Jul 27 17:30:29.359963 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.359956    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a27948-bc69-4336-b004-cf8c1455ecd0-kserve-provision-location\") pod \"22a27948-bc69-4336-b004-cf8c1455ecd0\" (UID: \"22a27948-bc69-4336-b004-cf8c1455ecd0\") "
Jul 27 17:30:29.360424 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.360370    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a27948-bc69-4336-b004-cf8c1455ecd0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22a27948-bc69-4336-b004-cf8c1455ecd0" (UID: "22a27948-bc69-4336-b004-cf8c1455ecd0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:30:29.362568 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.362528    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a27948-bc69-4336-b004-cf8c1455ecd0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22a27948-bc69-4336-b004-cf8c1455ecd0" (UID: "22a27948-bc69-4336-b004-cf8c1455ecd0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:30:29.369947 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.369911    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"]
Jul 27 17:30:29.375935 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.375678    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-n7pl7"]
Jul 27 17:30:29.437791 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.437747    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a27948-bc69-4336-b004-cf8c1455ecd0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "22a27948-bc69-4336-b004-cf8c1455ecd0" (UID: "22a27948-bc69-4336-b004-cf8c1455ecd0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:30:29.461339 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.461298    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a27948-bc69-4336-b004-cf8c1455ecd0-kserve-provision-location\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:30:29.461339 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.461340    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a27948-bc69-4336-b004-cf8c1455ecd0-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:30:29.461582 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:29.461355    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22a27948-bc69-4336-b004-cf8c1455ecd0-tmp-dir\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:30:30.051884 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:30.051793    2566 generic.go:356] "Generic (PLEG): container finished" podID="22a27948-bc69-4336-b004-cf8c1455ecd0" containerID="55bb3128b52c39c2b0a3059694d3e1bb6015dd2875f61dab2bbc926dcf4ae77e" exitCode=0
Jul 27 17:30:30.051884 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:30.051865    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"
Jul 27 17:30:30.052085 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:30.051879    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx" event={"ID":"22a27948-bc69-4336-b004-cf8c1455ecd0","Type":"ContainerDied","Data":"55bb3128b52c39c2b0a3059694d3e1bb6015dd2875f61dab2bbc926dcf4ae77e"}
Jul 27 17:30:30.052085 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:30.051988    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx" event={"ID":"22a27948-bc69-4336-b004-cf8c1455ecd0","Type":"ContainerDied","Data":"a393e2ba9ea0286b43bf7122a84be3618336749ed8d41cba4e5edfe4952cd352"}
Jul 27 17:30:30.052085 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:30.052023    2566 scope.go:117] "RemoveContainer" containerID="55bb3128b52c39c2b0a3059694d3e1bb6015dd2875f61dab2bbc926dcf4ae77e"
Jul 27 17:30:30.061559 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:30.061542    2566 scope.go:117] "RemoveContainer" containerID="ad9a9a0c68a1e95e1fdeaaa84a69e79e1e964db9dc65137eb2a98077552ac05a"
Jul 27 17:30:30.070148 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:30.070129    2566 scope.go:117] "RemoveContainer" containerID="55bb3128b52c39c2b0a3059694d3e1bb6015dd2875f61dab2bbc926dcf4ae77e"
Jul 27 17:30:30.070419 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:30:30.070401    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55bb3128b52c39c2b0a3059694d3e1bb6015dd2875f61dab2bbc926dcf4ae77e\": container with ID starting with 55bb3128b52c39c2b0a3059694d3e1bb6015dd2875f61dab2bbc926dcf4ae77e not found: ID does not exist" containerID="55bb3128b52c39c2b0a3059694d3e1bb6015dd2875f61dab2bbc926dcf4ae77e"
Jul 27 17:30:30.070466 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:30.070425    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55bb3128b52c39c2b0a3059694d3e1bb6015dd2875f61dab2bbc926dcf4ae77e"} err="failed to get container status \"55bb3128b52c39c2b0a3059694d3e1bb6015dd2875f61dab2bbc926dcf4ae77e\": rpc error: code = NotFound desc = could not find container \"55bb3128b52c39c2b0a3059694d3e1bb6015dd2875f61dab2bbc926dcf4ae77e\": container with ID starting with 55bb3128b52c39c2b0a3059694d3e1bb6015dd2875f61dab2bbc926dcf4ae77e not found: ID does not exist"
Jul 27 17:30:30.070466 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:30.070442    2566 scope.go:117] "RemoveContainer" containerID="ad9a9a0c68a1e95e1fdeaaa84a69e79e1e964db9dc65137eb2a98077552ac05a"
Jul 27 17:30:30.070652 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:30:30.070630    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad9a9a0c68a1e95e1fdeaaa84a69e79e1e964db9dc65137eb2a98077552ac05a\": container with ID starting with ad9a9a0c68a1e95e1fdeaaa84a69e79e1e964db9dc65137eb2a98077552ac05a not found: ID does not exist" containerID="ad9a9a0c68a1e95e1fdeaaa84a69e79e1e964db9dc65137eb2a98077552ac05a"
Jul 27 17:30:30.070710 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:30.070662    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad9a9a0c68a1e95e1fdeaaa84a69e79e1e964db9dc65137eb2a98077552ac05a"} err="failed to get container status \"ad9a9a0c68a1e95e1fdeaaa84a69e79e1e964db9dc65137eb2a98077552ac05a\": rpc error: code = NotFound desc = could not find container \"ad9a9a0c68a1e95e1fdeaaa84a69e79e1e964db9dc65137eb2a98077552ac05a\": container with ID starting with ad9a9a0c68a1e95e1fdeaaa84a69e79e1e964db9dc65137eb2a98077552ac05a not found: ID does not exist"
Jul 27 17:30:30.075459 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:30.075437    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"]
Jul 27 17:30:30.077586 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:30.077565    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69f5c7c9cc-nx6cx"]
Jul 27 17:30:30.403830 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:30.403749    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22a27948-bc69-4336-b004-cf8c1455ecd0" path="/var/lib/kubelet/pods/22a27948-bc69-4336-b004-cf8c1455ecd0/volumes"
Jul 27 17:30:30.404226 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:30.404194    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a" path="/var/lib/kubelet/pods/c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a/volumes"
Jul 27 17:30:57.465958 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:57.465923    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:30:57.472736 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:30:57.472710    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:31:07.710715 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.710680    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"]
Jul 27 17:31:07.711114 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.711045    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a" containerName="storage-initializer"
Jul 27 17:31:07.711114 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.711056    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a" containerName="storage-initializer"
Jul 27 17:31:07.711114 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.711069    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a27948-bc69-4336-b004-cf8c1455ecd0" containerName="storage-initializer"
Jul 27 17:31:07.711114 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.711075    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a27948-bc69-4336-b004-cf8c1455ecd0" containerName="storage-initializer"
Jul 27 17:31:07.711114 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.711081    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a" containerName="main"
Jul 27 17:31:07.711114 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.711087    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a" containerName="main"
Jul 27 17:31:07.711114 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.711103    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a27948-bc69-4336-b004-cf8c1455ecd0" containerName="main"
Jul 27 17:31:07.711114 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.711109    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a27948-bc69-4336-b004-cf8c1455ecd0" containerName="main"
Jul 27 17:31:07.711393 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.711207    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6131eb4-cf32-4b03-ba85-9fb59c5e5b0a" containerName="main"
Jul 27 17:31:07.711393 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.711216    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a27948-bc69-4336-b004-cf8c1455ecd0" containerName="main"
Jul 27 17:31:07.714375 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.714324    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.716790 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.716766    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 27 17:31:07.716922 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.716826    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 27 17:31:07.716922 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.716913    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-4tfpm\""
Jul 27 17:31:07.717052 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.716924    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 27 17:31:07.717945 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.717922    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 27 17:31:07.723863 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.723826    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"]
Jul 27 17:31:07.801403 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.801366    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.801403 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.801409    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-home\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.801694 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.801427    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fdf2906-f939-4179-8b6c-a7bd43695652-tls-certs\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.801694 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.801525    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-dshm\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.801694 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.801552    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-model-cache\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.801694 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.801582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.801694 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.801614    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnw2d\" (UniqueName: \"kubernetes.io/projected/2fdf2906-f939-4179-8b6c-a7bd43695652-kube-api-access-lnw2d\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.902009 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.901966    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.902009 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.902009    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnw2d\" (UniqueName: \"kubernetes.io/projected/2fdf2906-f939-4179-8b6c-a7bd43695652-kube-api-access-lnw2d\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.902257 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.902134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.902257 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.902216    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-home\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.902257 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.902248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fdf2906-f939-4179-8b6c-a7bd43695652-tls-certs\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.902423 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.902350    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-dshm\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.902423 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.902388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-model-cache\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.902423 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.902409    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.902594 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.902484    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.902675 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.902653    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-home\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.902727 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.902696    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-model-cache\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.904510 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.904489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-dshm\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.904775 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.904756    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fdf2906-f939-4179-8b6c-a7bd43695652-tls-certs\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.910747 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.910721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnw2d\" (UniqueName: \"kubernetes.io/projected/2fdf2906-f939-4179-8b6c-a7bd43695652-kube-api-access-lnw2d\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:07.943447 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.943411    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"]
Jul 27 17:31:07.947404 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.947373    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"
Jul 27 17:31:07.957987 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:07.957958    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"]
Jul 27 17:31:08.002815 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:08.002721    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aab27a7-72a4-4d73-9e46-619acfb80b8f-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-775958968c-n4tbr\" (UID: \"4aab27a7-72a4-4d73-9e46-619acfb80b8f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"
Jul 27 17:31:08.002962 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:08.002843    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aab27a7-72a4-4d73-9e46-619acfb80b8f-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-775958968c-n4tbr\" (UID: \"4aab27a7-72a4-4d73-9e46-619acfb80b8f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"
Jul 27 17:31:08.002962 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:08.002886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aab27a7-72a4-4d73-9e46-619acfb80b8f-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-775958968c-n4tbr\" (UID: \"4aab27a7-72a4-4d73-9e46-619acfb80b8f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"
Jul 27 17:31:08.026488 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:08.026448    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:08.103578 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:08.103543    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aab27a7-72a4-4d73-9e46-619acfb80b8f-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-775958968c-n4tbr\" (UID: \"4aab27a7-72a4-4d73-9e46-619acfb80b8f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"
Jul 27 17:31:08.103767 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:08.103631    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aab27a7-72a4-4d73-9e46-619acfb80b8f-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-775958968c-n4tbr\" (UID: \"4aab27a7-72a4-4d73-9e46-619acfb80b8f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"
Jul 27 17:31:08.103767 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:08.103659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aab27a7-72a4-4d73-9e46-619acfb80b8f-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-775958968c-n4tbr\" (UID: \"4aab27a7-72a4-4d73-9e46-619acfb80b8f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"
Jul 27 17:31:08.103974 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:08.103953    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aab27a7-72a4-4d73-9e46-619acfb80b8f-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-775958968c-n4tbr\" (UID: \"4aab27a7-72a4-4d73-9e46-619acfb80b8f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"
Jul 27 17:31:08.104135 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:08.104120    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aab27a7-72a4-4d73-9e46-619acfb80b8f-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-775958968c-n4tbr\" (UID: \"4aab27a7-72a4-4d73-9e46-619acfb80b8f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"
Jul 27 17:31:08.106243 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:08.106219    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aab27a7-72a4-4d73-9e46-619acfb80b8f-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-775958968c-n4tbr\" (UID: \"4aab27a7-72a4-4d73-9e46-619acfb80b8f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"
Jul 27 17:31:08.172972 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:08.172945    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"]
Jul 27 17:31:08.174524 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:31:08.174497    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fdf2906_f939_4179_8b6c_a7bd43695652.slice/crio-62a79886378d226202901471a551b7cb95643bd5dc06fe07ed62da82576c5938 WatchSource:0}: Error finding container 62a79886378d226202901471a551b7cb95643bd5dc06fe07ed62da82576c5938: Status 404 returned error can't find the container with id 62a79886378d226202901471a551b7cb95643bd5dc06fe07ed62da82576c5938
Jul 27 17:31:08.260817 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:08.260727    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"
Jul 27 17:31:08.280453 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:08.280414    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7" event={"ID":"2fdf2906-f939-4179-8b6c-a7bd43695652","Type":"ContainerStarted","Data":"be49773f8351f9a7ad1a49e3a5d187949adfe7a4407507efab54347ae658fafa"}
Jul 27 17:31:08.280607 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:08.280478    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7" event={"ID":"2fdf2906-f939-4179-8b6c-a7bd43695652","Type":"ContainerStarted","Data":"62a79886378d226202901471a551b7cb95643bd5dc06fe07ed62da82576c5938"}
Jul 27 17:31:08.404515 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:31:08.404485    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aab27a7_72a4_4d73_9e46_619acfb80b8f.slice/crio-0606d0ed5df503ba6eedc2f6141ee78c019678b1d57bea7ef7c4facc7720cf9b WatchSource:0}: Error finding container 0606d0ed5df503ba6eedc2f6141ee78c019678b1d57bea7ef7c4facc7720cf9b: Status 404 returned error can't find the container with id 0606d0ed5df503ba6eedc2f6141ee78c019678b1d57bea7ef7c4facc7720cf9b
Jul 27 17:31:08.404776 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:08.404755    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"]
Jul 27 17:31:09.296992 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:09.296947    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr" event={"ID":"4aab27a7-72a4-4d73-9e46-619acfb80b8f","Type":"ContainerStarted","Data":"a341fad342b20c917b7010b553fd1a84d2a2ee296a403a51a94c9a7f7fce0a3f"}
Jul 27 17:31:09.297573 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:09.297017    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr" event={"ID":"4aab27a7-72a4-4d73-9e46-619acfb80b8f","Type":"ContainerStarted","Data":"0606d0ed5df503ba6eedc2f6141ee78c019678b1d57bea7ef7c4facc7720cf9b"}
Jul 27 17:31:17.347321 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:17.347287    2566 generic.go:356] "Generic (PLEG): container finished" podID="4aab27a7-72a4-4d73-9e46-619acfb80b8f" containerID="a341fad342b20c917b7010b553fd1a84d2a2ee296a403a51a94c9a7f7fce0a3f" exitCode=0
Jul 27 17:31:17.347690 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:17.347370    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr" event={"ID":"4aab27a7-72a4-4d73-9e46-619acfb80b8f","Type":"ContainerDied","Data":"a341fad342b20c917b7010b553fd1a84d2a2ee296a403a51a94c9a7f7fce0a3f"}
Jul 27 17:31:18.355347 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:18.355249    2566 generic.go:356] "Generic (PLEG): container finished" podID="2fdf2906-f939-4179-8b6c-a7bd43695652" containerID="be49773f8351f9a7ad1a49e3a5d187949adfe7a4407507efab54347ae658fafa" exitCode=0
Jul 27 17:31:18.355347 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:18.355325    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7" event={"ID":"2fdf2906-f939-4179-8b6c-a7bd43695652","Type":"ContainerDied","Data":"be49773f8351f9a7ad1a49e3a5d187949adfe7a4407507efab54347ae658fafa"}
Jul 27 17:31:18.357524 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:18.357497    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr" event={"ID":"4aab27a7-72a4-4d73-9e46-619acfb80b8f","Type":"ContainerStarted","Data":"e468d100ed83cf68d698b5d4cc46ac70400d9977a23fd20b38228cee199767a2"}
Jul 27 17:31:18.357745 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:18.357724    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"
Jul 27 17:31:18.392113 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:18.392062    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr" podStartSLOduration=2.601078236 podStartE2EDuration="11.392046828s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.790968592s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:31:18.389281177 +0000 UTC m=+1822.464545655" watchObservedRunningTime="2026-07-27 17:31:18.392046828 +0000 UTC m=+1822.467311315"
Jul 27 17:31:19.369147 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:19.369104    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7" event={"ID":"2fdf2906-f939-4179-8b6c-a7bd43695652","Type":"ContainerStarted","Data":"f91dfc4f2e4e798b723eda1a4b3d689c22d4d2781d7ae1aa184a056939e8666f"}
Jul 27 17:31:19.388807 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:19.388742    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7" podStartSLOduration=2.616156714 podStartE2EDuration="12.388724238s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.772567524s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:31:19.387992922 +0000 UTC m=+1823.463257410" watchObservedRunningTime="2026-07-27 17:31:19.388724238 +0000 UTC m=+1823.463988725"
Jul 27 17:31:20.026794 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:20.026761    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:28.027348 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:28.027306    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:28.045984 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:28.045961    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:28.443084 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:28.443060    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:49.381299 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:49.381263    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"
Jul 27 17:31:50.804718 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:50.804682    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-775958968c-n4tbr_4aab27a7-72a4-4d73-9e46-619acfb80b8f/main/0.log"
Jul 27 17:31:51.041005 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.040972    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"]
Jul 27 17:31:51.046647 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.046619    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"]
Jul 27 17:31:51.046835 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.046808    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr" podUID="4aab27a7-72a4-4d73-9e46-619acfb80b8f" containerName="main" containerID="cri-o://e468d100ed83cf68d698b5d4cc46ac70400d9977a23fd20b38228cee199767a2" gracePeriod=30
Jul 27 17:31:51.050727 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:31:51.050698    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:31:51.050727 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:31:51.050727 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:31:51.050727 ip-10-0-143-86 kubenswrapper[2566]:  > containerID="f91dfc4f2e4e798b723eda1a4b3d689c22d4d2781d7ae1aa184a056939e8666f" cmd=["/bin/sleep","15"]
Jul 27 17:31:51.050930 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.050744    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7" message="" err=<
Jul 27 17:31:51.050930 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:31:51.050930 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:31:51.050930 ip-10-0-143-86 kubenswrapper[2566]:  >
Jul 27 17:31:51.050930 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:31:51.050755    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:31:51.050930 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:31:51.050930 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:31:51.050930 ip-10-0-143-86 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7" podUID="2fdf2906-f939-4179-8b6c-a7bd43695652" containerName="main" containerID="cri-o://f91dfc4f2e4e798b723eda1a4b3d689c22d4d2781d7ae1aa184a056939e8666f"
Jul 27 17:31:51.050930 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.050797    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7" podUID="2fdf2906-f939-4179-8b6c-a7bd43695652" containerName="main" containerID="cri-o://f91dfc4f2e4e798b723eda1a4b3d689c22d4d2781d7ae1aa184a056939e8666f" gracePeriod=60
Jul 27 17:31:51.412215 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.412123    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:51.522929 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.522895    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-model-cache\") pod \"2fdf2906-f939-4179-8b6c-a7bd43695652\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") "
Jul 27 17:31:51.523090 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.522945    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-tmp-dir\") pod \"2fdf2906-f939-4179-8b6c-a7bd43695652\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") "
Jul 27 17:31:51.523090 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.522980    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-dshm\") pod \"2fdf2906-f939-4179-8b6c-a7bd43695652\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") "
Jul 27 17:31:51.523090 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.523011    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnw2d\" (UniqueName: \"kubernetes.io/projected/2fdf2906-f939-4179-8b6c-a7bd43695652-kube-api-access-lnw2d\") pod \"2fdf2906-f939-4179-8b6c-a7bd43695652\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") "
Jul 27 17:31:51.523090 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.523037    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-home\") pod \"2fdf2906-f939-4179-8b6c-a7bd43695652\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") "
Jul 27 17:31:51.523090 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.523054    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fdf2906-f939-4179-8b6c-a7bd43695652-tls-certs\") pod \"2fdf2906-f939-4179-8b6c-a7bd43695652\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") "
Jul 27 17:31:51.523404 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.523208    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-model-cache" (OuterVolumeSpecName: "model-cache") pod "2fdf2906-f939-4179-8b6c-a7bd43695652" (UID: "2fdf2906-f939-4179-8b6c-a7bd43695652"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:31:51.523404 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.523221    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-kserve-provision-location\") pod \"2fdf2906-f939-4179-8b6c-a7bd43695652\" (UID: \"2fdf2906-f939-4179-8b6c-a7bd43695652\") "
Jul 27 17:31:51.523404 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.523292    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2fdf2906-f939-4179-8b6c-a7bd43695652" (UID: "2fdf2906-f939-4179-8b6c-a7bd43695652"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:31:51.523404 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.523341    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-home" (OuterVolumeSpecName: "home") pod "2fdf2906-f939-4179-8b6c-a7bd43695652" (UID: "2fdf2906-f939-4179-8b6c-a7bd43695652"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:31:51.523576 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.523556    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-model-cache\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:31:51.523622 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.523583    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-tmp-dir\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:31:51.523622 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.523595    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-home\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:31:51.525885 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.525835    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-dshm" (OuterVolumeSpecName: "dshm") pod "2fdf2906-f939-4179-8b6c-a7bd43695652" (UID: "2fdf2906-f939-4179-8b6c-a7bd43695652"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:31:51.526016 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.525877    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fdf2906-f939-4179-8b6c-a7bd43695652-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fdf2906-f939-4179-8b6c-a7bd43695652" (UID: "2fdf2906-f939-4179-8b6c-a7bd43695652"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:51.526016 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.525888    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fdf2906-f939-4179-8b6c-a7bd43695652-kube-api-access-lnw2d" (OuterVolumeSpecName: "kube-api-access-lnw2d") pod "2fdf2906-f939-4179-8b6c-a7bd43695652" (UID: "2fdf2906-f939-4179-8b6c-a7bd43695652"). InnerVolumeSpecName "kube-api-access-lnw2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:31:51.567115 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.567088    2566 generic.go:356] "Generic (PLEG): container finished" podID="2fdf2906-f939-4179-8b6c-a7bd43695652" containerID="f91dfc4f2e4e798b723eda1a4b3d689c22d4d2781d7ae1aa184a056939e8666f" exitCode=0
Jul 27 17:31:51.567268 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.567138    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"
Jul 27 17:31:51.567268 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.567184    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7" event={"ID":"2fdf2906-f939-4179-8b6c-a7bd43695652","Type":"ContainerDied","Data":"f91dfc4f2e4e798b723eda1a4b3d689c22d4d2781d7ae1aa184a056939e8666f"}
Jul 27 17:31:51.567268 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.567243    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7" event={"ID":"2fdf2906-f939-4179-8b6c-a7bd43695652","Type":"ContainerDied","Data":"62a79886378d226202901471a551b7cb95643bd5dc06fe07ed62da82576c5938"}
Jul 27 17:31:51.567425 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.567276    2566 scope.go:117] "RemoveContainer" containerID="f91dfc4f2e4e798b723eda1a4b3d689c22d4d2781d7ae1aa184a056939e8666f"
Jul 27 17:31:51.569426 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.569387    2566 generic.go:356] "Generic (PLEG): container finished" podID="4aab27a7-72a4-4d73-9e46-619acfb80b8f" containerID="e468d100ed83cf68d698b5d4cc46ac70400d9977a23fd20b38228cee199767a2" exitCode=0
Jul 27 17:31:51.569525 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.569449    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr" event={"ID":"4aab27a7-72a4-4d73-9e46-619acfb80b8f","Type":"ContainerDied","Data":"e468d100ed83cf68d698b5d4cc46ac70400d9977a23fd20b38228cee199767a2"}
Jul 27 17:31:51.578918 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.578892    2566 scope.go:117] "RemoveContainer" containerID="be49773f8351f9a7ad1a49e3a5d187949adfe7a4407507efab54347ae658fafa"
Jul 27 17:31:51.591692 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.591669    2566 scope.go:117] "RemoveContainer" containerID="f91dfc4f2e4e798b723eda1a4b3d689c22d4d2781d7ae1aa184a056939e8666f"
Jul 27 17:31:51.592039 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:31:51.592013    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91dfc4f2e4e798b723eda1a4b3d689c22d4d2781d7ae1aa184a056939e8666f\": container with ID starting with f91dfc4f2e4e798b723eda1a4b3d689c22d4d2781d7ae1aa184a056939e8666f not found: ID does not exist" containerID="f91dfc4f2e4e798b723eda1a4b3d689c22d4d2781d7ae1aa184a056939e8666f"
Jul 27 17:31:51.592130 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.592054    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91dfc4f2e4e798b723eda1a4b3d689c22d4d2781d7ae1aa184a056939e8666f"} err="failed to get container status \"f91dfc4f2e4e798b723eda1a4b3d689c22d4d2781d7ae1aa184a056939e8666f\": rpc error: code = NotFound desc = could not find container \"f91dfc4f2e4e798b723eda1a4b3d689c22d4d2781d7ae1aa184a056939e8666f\": container with ID starting with f91dfc4f2e4e798b723eda1a4b3d689c22d4d2781d7ae1aa184a056939e8666f not found: ID does not exist"
Jul 27 17:31:51.592130 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.592080    2566 scope.go:117] "RemoveContainer" containerID="be49773f8351f9a7ad1a49e3a5d187949adfe7a4407507efab54347ae658fafa"
Jul 27 17:31:51.592482 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:31:51.592452    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be49773f8351f9a7ad1a49e3a5d187949adfe7a4407507efab54347ae658fafa\": container with ID starting with be49773f8351f9a7ad1a49e3a5d187949adfe7a4407507efab54347ae658fafa not found: ID does not exist" containerID="be49773f8351f9a7ad1a49e3a5d187949adfe7a4407507efab54347ae658fafa"
Jul 27 17:31:51.592577 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.592492    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be49773f8351f9a7ad1a49e3a5d187949adfe7a4407507efab54347ae658fafa"} err="failed to get container status \"be49773f8351f9a7ad1a49e3a5d187949adfe7a4407507efab54347ae658fafa\": rpc error: code = NotFound desc = could not find container \"be49773f8351f9a7ad1a49e3a5d187949adfe7a4407507efab54347ae658fafa\": container with ID starting with be49773f8351f9a7ad1a49e3a5d187949adfe7a4407507efab54347ae658fafa not found: ID does not exist"
Jul 27 17:31:51.624853 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.624820    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnw2d\" (UniqueName: \"kubernetes.io/projected/2fdf2906-f939-4179-8b6c-a7bd43695652-kube-api-access-lnw2d\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:31:51.624853 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.624855    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fdf2906-f939-4179-8b6c-a7bd43695652-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:31:51.625025 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.624869    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-dshm\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:31:51.702550 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.702520    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"
Jul 27 17:31:51.725128 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.725093    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aab27a7-72a4-4d73-9e46-619acfb80b8f-tmp-dir\") pod \"4aab27a7-72a4-4d73-9e46-619acfb80b8f\" (UID: \"4aab27a7-72a4-4d73-9e46-619acfb80b8f\") "
Jul 27 17:31:51.725322 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.725242    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aab27a7-72a4-4d73-9e46-619acfb80b8f-kserve-provision-location\") pod \"4aab27a7-72a4-4d73-9e46-619acfb80b8f\" (UID: \"4aab27a7-72a4-4d73-9e46-619acfb80b8f\") "
Jul 27 17:31:51.725322 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.725315    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aab27a7-72a4-4d73-9e46-619acfb80b8f-tls-certs\") pod \"4aab27a7-72a4-4d73-9e46-619acfb80b8f\" (UID: \"4aab27a7-72a4-4d73-9e46-619acfb80b8f\") "
Jul 27 17:31:51.725666 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.725636    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aab27a7-72a4-4d73-9e46-619acfb80b8f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4aab27a7-72a4-4d73-9e46-619acfb80b8f" (UID: "4aab27a7-72a4-4d73-9e46-619acfb80b8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:31:51.728221 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.728190    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aab27a7-72a4-4d73-9e46-619acfb80b8f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4aab27a7-72a4-4d73-9e46-619acfb80b8f" (UID: "4aab27a7-72a4-4d73-9e46-619acfb80b8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:51.826493 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.826447    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aab27a7-72a4-4d73-9e46-619acfb80b8f-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:31:51.826493 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.826486    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aab27a7-72a4-4d73-9e46-619acfb80b8f-tmp-dir\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:31:51.841631 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.841588    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2fdf2906-f939-4179-8b6c-a7bd43695652" (UID: "2fdf2906-f939-4179-8b6c-a7bd43695652"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:31:51.897604 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.897571    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"]
Jul 27 17:31:51.902801 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.902772    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-ct8f7"]
Jul 27 17:31:51.927096 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:51.927057    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fdf2906-f939-4179-8b6c-a7bd43695652-kserve-provision-location\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:31:52.161143 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:52.161087    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aab27a7-72a4-4d73-9e46-619acfb80b8f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4aab27a7-72a4-4d73-9e46-619acfb80b8f" (UID: "4aab27a7-72a4-4d73-9e46-619acfb80b8f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:31:52.230004 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:52.229963    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4aab27a7-72a4-4d73-9e46-619acfb80b8f-kserve-provision-location\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:31:52.403280 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:52.403247    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fdf2906-f939-4179-8b6c-a7bd43695652" path="/var/lib/kubelet/pods/2fdf2906-f939-4179-8b6c-a7bd43695652/volumes"
Jul 27 17:31:52.577412 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:52.577386    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"
Jul 27 17:31:52.577579 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:52.577380    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr" event={"ID":"4aab27a7-72a4-4d73-9e46-619acfb80b8f","Type":"ContainerDied","Data":"0606d0ed5df503ba6eedc2f6141ee78c019678b1d57bea7ef7c4facc7720cf9b"}
Jul 27 17:31:52.577579 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:52.577516    2566 scope.go:117] "RemoveContainer" containerID="e468d100ed83cf68d698b5d4cc46ac70400d9977a23fd20b38228cee199767a2"
Jul 27 17:31:52.586760 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:52.586741    2566 scope.go:117] "RemoveContainer" containerID="a341fad342b20c917b7010b553fd1a84d2a2ee296a403a51a94c9a7f7fce0a3f"
Jul 27 17:31:52.596267 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:52.596249    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"]
Jul 27 17:31:52.598406 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:52.598384    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-775958968c-n4tbr"]
Jul 27 17:31:54.403541 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:31:54.403508    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aab27a7-72a4-4d73-9e46-619acfb80b8f" path="/var/lib/kubelet/pods/4aab27a7-72a4-4d73-9e46-619acfb80b8f/volumes"
Jul 27 17:32:22.324375 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.324292    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"]
Jul 27 17:32:22.324814 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.324733    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aab27a7-72a4-4d73-9e46-619acfb80b8f" containerName="storage-initializer"
Jul 27 17:32:22.324814 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.324747    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aab27a7-72a4-4d73-9e46-619acfb80b8f" containerName="storage-initializer"
Jul 27 17:32:22.324814 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.324756    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aab27a7-72a4-4d73-9e46-619acfb80b8f" containerName="main"
Jul 27 17:32:22.324814 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.324762    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aab27a7-72a4-4d73-9e46-619acfb80b8f" containerName="main"
Jul 27 17:32:22.324814 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.324773    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fdf2906-f939-4179-8b6c-a7bd43695652" containerName="main"
Jul 27 17:32:22.324814 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.324778    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fdf2906-f939-4179-8b6c-a7bd43695652" containerName="main"
Jul 27 17:32:22.324814 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.324785    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fdf2906-f939-4179-8b6c-a7bd43695652" containerName="storage-initializer"
Jul 27 17:32:22.324814 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.324790    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fdf2906-f939-4179-8b6c-a7bd43695652" containerName="storage-initializer"
Jul 27 17:32:22.325275 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.324872    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aab27a7-72a4-4d73-9e46-619acfb80b8f" containerName="main"
Jul 27 17:32:22.325275 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.324884    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fdf2906-f939-4179-8b6c-a7bd43695652" containerName="main"
Jul 27 17:32:22.327984 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.327967    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.331430 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.331406    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 27 17:32:22.331548 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.331436    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 27 17:32:22.331548 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.331406    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 27 17:32:22.331548 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.331469    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-lpgtt\""
Jul 27 17:32:22.337338 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.337311    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"]
Jul 27 17:32:22.507559 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.507523    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.507737 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.507639    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.507737 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.507674    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.507737 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.507706    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.507874 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.507769    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6b8k\" (UniqueName: \"kubernetes.io/projected/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-kube-api-access-r6b8k\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.507874 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.507804    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.608564 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.608474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.608564 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.608534    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.608564 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.608559    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.608808 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.608577    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.608808 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.608612    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6b8k\" (UniqueName: \"kubernetes.io/projected/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-kube-api-access-r6b8k\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.608808 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.608740    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.608933 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.608908    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.609009 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.608987    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.609046 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.608994    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.611062 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.611031    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.611396 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.611370    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.616078 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.616054    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6b8k\" (UniqueName: \"kubernetes.io/projected/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-kube-api-access-r6b8k\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.639092 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.639065    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:22.780949 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:22.780918    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"]
Jul 27 17:32:22.783060 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:32:22.783031    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b97e413_afc0_4bee_afa1_4c98e8c5e07d.slice/crio-7e49970e799bbc4ec938bc93661b0eee82a4f470d5d5c82669dbb44222a8f022 WatchSource:0}: Error finding container 7e49970e799bbc4ec938bc93661b0eee82a4f470d5d5c82669dbb44222a8f022: Status 404 returned error can't find the container with id 7e49970e799bbc4ec938bc93661b0eee82a4f470d5d5c82669dbb44222a8f022
Jul 27 17:32:23.745452 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:23.745419    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48" event={"ID":"3b97e413-afc0-4bee-afa1-4c98e8c5e07d","Type":"ContainerStarted","Data":"e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323"}
Jul 27 17:32:23.745827 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:23.745471    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48" event={"ID":"3b97e413-afc0-4bee-afa1-4c98e8c5e07d","Type":"ContainerStarted","Data":"7e49970e799bbc4ec938bc93661b0eee82a4f470d5d5c82669dbb44222a8f022"}
Jul 27 17:32:23.769441 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:23.769389    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48" podStartSLOduration=1.769374027 podStartE2EDuration="1.769374027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:32:23.767520444 +0000 UTC m=+1887.842784935" watchObservedRunningTime="2026-07-27 17:32:23.769374027 +0000 UTC m=+1887.844638514"
Jul 27 17:32:24.639944 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:24.639908    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:32.639245 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:32.639201    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:32.652489 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:32.652450    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:32.803765 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:32.803738    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:56.216707 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.216669    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"]
Jul 27 17:32:56.228480 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:32:56.228440    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:32:56.228480 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:32:56.228480 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:32:56.228480 ip-10-0-143-86 kubenswrapper[2566]:  > containerID="e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323" cmd=["/bin/sleep","15"]
Jul 27 17:32:56.228805 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.228502    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48" message="" err=<
Jul 27 17:32:56.228805 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:32:56.228805 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:32:56.228805 ip-10-0-143-86 kubenswrapper[2566]:  >
Jul 27 17:32:56.228805 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:32:56.228519    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:32:56.228805 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:32:56.228805 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:32:56.228805 ip-10-0-143-86 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48" podUID="3b97e413-afc0-4bee-afa1-4c98e8c5e07d" containerName="main" containerID="cri-o://e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323"
Jul 27 17:32:56.228805 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.228588    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48" podUID="3b97e413-afc0-4bee-afa1-4c98e8c5e07d" containerName="main" containerID="cri-o://e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323" gracePeriod=60
Jul 27 17:32:56.568774 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.568750    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:56.722967 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.722929    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-model-cache\") pod \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") "
Jul 27 17:32:56.722967 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.722972    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6b8k\" (UniqueName: \"kubernetes.io/projected/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-kube-api-access-r6b8k\") pod \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") "
Jul 27 17:32:56.723233 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.722998    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-tls-certs\") pod \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") "
Jul 27 17:32:56.723233 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.723042    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-home\") pod \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") "
Jul 27 17:32:56.723233 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.723086    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-tmp-dir\") pod \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") "
Jul 27 17:32:56.723233 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.723132    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-dshm\") pod \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\" (UID: \"3b97e413-afc0-4bee-afa1-4c98e8c5e07d\") "
Jul 27 17:32:56.723450 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.723251    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b97e413-afc0-4bee-afa1-4c98e8c5e07d" (UID: "3b97e413-afc0-4bee-afa1-4c98e8c5e07d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:32:56.723450 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.723351    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-home" (OuterVolumeSpecName: "home") pod "3b97e413-afc0-4bee-afa1-4c98e8c5e07d" (UID: "3b97e413-afc0-4bee-afa1-4c98e8c5e07d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:32:56.723450 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.723416    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b97e413-afc0-4bee-afa1-4c98e8c5e07d" (UID: "3b97e413-afc0-4bee-afa1-4c98e8c5e07d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:32:56.723612 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.723480    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-home\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:32:56.723612 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.723502    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-model-cache\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:32:56.725588 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.725526    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-kube-api-access-r6b8k" (OuterVolumeSpecName: "kube-api-access-r6b8k") pod "3b97e413-afc0-4bee-afa1-4c98e8c5e07d" (UID: "3b97e413-afc0-4bee-afa1-4c98e8c5e07d"). InnerVolumeSpecName "kube-api-access-r6b8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:32:56.725588 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.725543    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-dshm" (OuterVolumeSpecName: "dshm") pod "3b97e413-afc0-4bee-afa1-4c98e8c5e07d" (UID: "3b97e413-afc0-4bee-afa1-4c98e8c5e07d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:32:56.725588 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.725529    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b97e413-afc0-4bee-afa1-4c98e8c5e07d" (UID: "3b97e413-afc0-4bee-afa1-4c98e8c5e07d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:32:56.824791 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.824701    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-tmp-dir\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:32:56.824791 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.824733    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-dshm\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:32:56.824791 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.824745    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6b8k\" (UniqueName: \"kubernetes.io/projected/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-kube-api-access-r6b8k\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:32:56.824791 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.824759    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b97e413-afc0-4bee-afa1-4c98e8c5e07d-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:32:56.924078 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.924043    2566 generic.go:356] "Generic (PLEG): container finished" podID="3b97e413-afc0-4bee-afa1-4c98e8c5e07d" containerID="e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323" exitCode=0
Jul 27 17:32:56.924261 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.924085    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"
Jul 27 17:32:56.924261 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.924101    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48" event={"ID":"3b97e413-afc0-4bee-afa1-4c98e8c5e07d","Type":"ContainerDied","Data":"e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323"}
Jul 27 17:32:56.924261 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.924143    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48" event={"ID":"3b97e413-afc0-4bee-afa1-4c98e8c5e07d","Type":"ContainerDied","Data":"7e49970e799bbc4ec938bc93661b0eee82a4f470d5d5c82669dbb44222a8f022"}
Jul 27 17:32:56.924261 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.924178    2566 scope.go:117] "RemoveContainer" containerID="e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323"
Jul 27 17:32:56.936503 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.936479    2566 scope.go:117] "RemoveContainer" containerID="e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323"
Jul 27 17:32:56.936856 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:32:56.936822    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323\": container with ID starting with e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323 not found: ID does not exist" containerID="e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323"
Jul 27 17:32:56.937195 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.936995    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323"} err="failed to get container status \"e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323\": rpc error: code = NotFound desc = could not find container \"e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323\": container with ID starting with e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323 not found: ID does not exist"
Jul 27 17:32:56.953056 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.953005    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"]
Jul 27 17:32:56.955564 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:56.955543    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tmt48"]
Jul 27 17:32:58.384403 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:32:58.384358    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b97e413_afc0_4bee_afa1_4c98e8c5e07d.slice/crio-conmon-e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323.scope\": RecentStats: unable to find data in memory cache]"
Jul 27 17:32:58.403441 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:32:58.403408    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b97e413-afc0-4bee-afa1-4c98e8c5e07d" path="/var/lib/kubelet/pods/3b97e413-afc0-4bee-afa1-4c98e8c5e07d/volumes"
Jul 27 17:33:02.783007 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:33:02.782972    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b97e413_afc0_4bee_afa1_4c98e8c5e07d.slice/crio-conmon-e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323.scope\": RecentStats: unable to find data in memory cache]"
Jul 27 17:33:12.830683 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:33:12.830642    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b97e413_afc0_4bee_afa1_4c98e8c5e07d.slice/crio-conmon-e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323.scope\": RecentStats: unable to find data in memory cache]"
Jul 27 17:33:13.341321 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:33:13.341280    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b97e413_afc0_4bee_afa1_4c98e8c5e07d.slice/crio-conmon-e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323.scope\": RecentStats: unable to find data in memory cache]"
Jul 27 17:33:22.882655 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:33:22.882620    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b97e413_afc0_4bee_afa1_4c98e8c5e07d.slice/crio-conmon-e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323.scope\": RecentStats: unable to find data in memory cache]"
Jul 27 17:33:23.194322 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.194290    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"]
Jul 27 17:33:23.194761 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.194739    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b97e413-afc0-4bee-afa1-4c98e8c5e07d" containerName="main"
Jul 27 17:33:23.194876 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.194764    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b97e413-afc0-4bee-afa1-4c98e8c5e07d" containerName="main"
Jul 27 17:33:23.194945 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.194894    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b97e413-afc0-4bee-afa1-4c98e8c5e07d" containerName="main"
Jul 27 17:33:23.198137 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.198117    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.202725 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.202699    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 27 17:33:23.204084 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.204059    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 27 17:33:23.204351 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.204103    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-9lqrd\""
Jul 27 17:33:23.204506 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.204484    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 27 17:33:23.207330 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.207306    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"]
Jul 27 17:33:23.247193 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.247128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m96vs\" (UniqueName: \"kubernetes.io/projected/ce66e16f-ca02-4e5b-af02-7034919579fe-kube-api-access-m96vs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.247394 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.247243    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.247394 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.247294    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce66e16f-ca02-4e5b-af02-7034919579fe-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.247394 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.247315    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.247577 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.247419    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.247577 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.247461    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.347983 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.347945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m96vs\" (UniqueName: \"kubernetes.io/projected/ce66e16f-ca02-4e5b-af02-7034919579fe-kube-api-access-m96vs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.348245 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.348000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.348245 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.348031    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce66e16f-ca02-4e5b-af02-7034919579fe-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.348245 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.348133    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.348245 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.348221    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.348511 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.348249    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.348511 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.348488    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.348624 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.348566    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.348624 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.348608    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.350388 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.350363    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.350764 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.350746    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce66e16f-ca02-4e5b-af02-7034919579fe-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.360638 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.360608    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m96vs\" (UniqueName: \"kubernetes.io/projected/ce66e16f-ca02-4e5b-af02-7034919579fe-kube-api-access-m96vs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.509262 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.509138    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:23.649084 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:23.649054    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"]
Jul 27 17:33:23.651112 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:33:23.651085    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce66e16f_ca02_4e5b_af02_7034919579fe.slice/crio-12fb4d91e7a9980dfe05472583969f5783d3f7a25367ee6b96679d5ec53ded30 WatchSource:0}: Error finding container 12fb4d91e7a9980dfe05472583969f5783d3f7a25367ee6b96679d5ec53ded30: Status 404 returned error can't find the container with id 12fb4d91e7a9980dfe05472583969f5783d3f7a25367ee6b96679d5ec53ded30
Jul 27 17:33:24.068542 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:24.068497    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5" event={"ID":"ce66e16f-ca02-4e5b-af02-7034919579fe","Type":"ContainerStarted","Data":"ca4616245e086115fe5db063c95777890c006a01e56091472d8ce8fd40d1dcca"}
Jul 27 17:33:24.068962 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:24.068565    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5" event={"ID":"ce66e16f-ca02-4e5b-af02-7034919579fe","Type":"ContainerStarted","Data":"12fb4d91e7a9980dfe05472583969f5783d3f7a25367ee6b96679d5ec53ded30"}
Jul 27 17:33:24.086060 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:24.086001    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5" podStartSLOduration=1.085986214 podStartE2EDuration="1.085986214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:33:24.084376311 +0000 UTC m=+1948.159640798" watchObservedRunningTime="2026-07-27 17:33:24.085986214 +0000 UTC m=+1948.161250757"
Jul 27 17:33:24.509347 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:24.509305    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:28.390126 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:33:28.390092    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b97e413_afc0_4bee_afa1_4c98e8c5e07d.slice/crio-conmon-e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323.scope\": RecentStats: unable to find data in memory cache]"
Jul 27 17:33:32.894502 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:33:32.894458    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b97e413_afc0_4bee_afa1_4c98e8c5e07d.slice/crio-conmon-e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323.scope\": RecentStats: unable to find data in memory cache]"
Jul 27 17:33:33.510035 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:33.509996    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:33.522921 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:33.522891    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:34.132370 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:33:34.132334    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:33:42.943446 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:33:42.943413    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b97e413_afc0_4bee_afa1_4c98e8c5e07d.slice/crio-conmon-e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323.scope\": RecentStats: unable to find data in memory cache]"
Jul 27 17:33:43.339544 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:33:43.339444    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b97e413_afc0_4bee_afa1_4c98e8c5e07d.slice/crio-conmon-e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323.scope\": RecentStats: unable to find data in memory cache]"
Jul 27 17:33:52.839820 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:33:52.839773    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b97e413_afc0_4bee_afa1_4c98e8c5e07d.slice/crio-conmon-e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323.scope\": RecentStats: unable to find data in memory cache]"
Jul 27 17:33:52.840217 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:33:52.840015    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b97e413_afc0_4bee_afa1_4c98e8c5e07d.slice/crio-conmon-e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323.scope\": RecentStats: unable to find data in memory cache]"
Jul 27 17:33:52.957787 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:33:52.957744    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b97e413_afc0_4bee_afa1_4c98e8c5e07d.slice/crio-conmon-e6ebe2b051296cca8ff2ce95ec8afb55b51873031a036d359cdbcc2596314323.scope\": RecentStats: unable to find data in memory cache]"
Jul 27 17:35:01.364845 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.364810    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"]
Jul 27 17:35:01.374427 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:35:01.374393    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:35:01.374427 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:35:01.374427 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:35:01.374427 ip-10-0-143-86 kubenswrapper[2566]:  > containerID="ca4616245e086115fe5db063c95777890c006a01e56091472d8ce8fd40d1dcca" cmd=["/bin/sleep","15"]
Jul 27 17:35:01.374721 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.374449    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5" message="" err=<
Jul 27 17:35:01.374721 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:35:01.374721 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:35:01.374721 ip-10-0-143-86 kubenswrapper[2566]:  >
Jul 27 17:35:01.374721 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:35:01.374465    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:35:01.374721 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:35:01.374721 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:35:01.374721 ip-10-0-143-86 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5" podUID="ce66e16f-ca02-4e5b-af02-7034919579fe" containerName="main" containerID="cri-o://ca4616245e086115fe5db063c95777890c006a01e56091472d8ce8fd40d1dcca"
Jul 27 17:35:01.374721 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.374506    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5" podUID="ce66e16f-ca02-4e5b-af02-7034919579fe" containerName="main" containerID="cri-o://ca4616245e086115fe5db063c95777890c006a01e56091472d8ce8fd40d1dcca" gracePeriod=60
Jul 27 17:35:01.620377 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.620279    2566 generic.go:356] "Generic (PLEG): container finished" podID="ce66e16f-ca02-4e5b-af02-7034919579fe" containerID="ca4616245e086115fe5db063c95777890c006a01e56091472d8ce8fd40d1dcca" exitCode=0
Jul 27 17:35:01.620377 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.620332    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5" event={"ID":"ce66e16f-ca02-4e5b-af02-7034919579fe","Type":"ContainerDied","Data":"ca4616245e086115fe5db063c95777890c006a01e56091472d8ce8fd40d1dcca"}
Jul 27 17:35:01.728201 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.728159    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:35:01.825685 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.825645    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m96vs\" (UniqueName: \"kubernetes.io/projected/ce66e16f-ca02-4e5b-af02-7034919579fe-kube-api-access-m96vs\") pod \"ce66e16f-ca02-4e5b-af02-7034919579fe\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") "
Jul 27 17:35:01.825685 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.825688    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-tmp-dir\") pod \"ce66e16f-ca02-4e5b-af02-7034919579fe\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") "
Jul 27 17:35:01.825951 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.825717    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce66e16f-ca02-4e5b-af02-7034919579fe-tls-certs\") pod \"ce66e16f-ca02-4e5b-af02-7034919579fe\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") "
Jul 27 17:35:01.825951 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.825738    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-dshm\") pod \"ce66e16f-ca02-4e5b-af02-7034919579fe\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") "
Jul 27 17:35:01.825951 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.825786    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-home\") pod \"ce66e16f-ca02-4e5b-af02-7034919579fe\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") "
Jul 27 17:35:01.825951 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.825819    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-model-cache\") pod \"ce66e16f-ca02-4e5b-af02-7034919579fe\" (UID: \"ce66e16f-ca02-4e5b-af02-7034919579fe\") "
Jul 27 17:35:01.826149 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.825993    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce66e16f-ca02-4e5b-af02-7034919579fe" (UID: "ce66e16f-ca02-4e5b-af02-7034919579fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:35:01.826149 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.826138    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-tmp-dir\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:35:01.826503 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.826471    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-home" (OuterVolumeSpecName: "home") pod "ce66e16f-ca02-4e5b-af02-7034919579fe" (UID: "ce66e16f-ca02-4e5b-af02-7034919579fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:35:01.826690 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.826646    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce66e16f-ca02-4e5b-af02-7034919579fe" (UID: "ce66e16f-ca02-4e5b-af02-7034919579fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:35:01.828643 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.828613    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-dshm" (OuterVolumeSpecName: "dshm") pod "ce66e16f-ca02-4e5b-af02-7034919579fe" (UID: "ce66e16f-ca02-4e5b-af02-7034919579fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:35:01.828744 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.828664    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce66e16f-ca02-4e5b-af02-7034919579fe-kube-api-access-m96vs" (OuterVolumeSpecName: "kube-api-access-m96vs") pod "ce66e16f-ca02-4e5b-af02-7034919579fe" (UID: "ce66e16f-ca02-4e5b-af02-7034919579fe"). InnerVolumeSpecName "kube-api-access-m96vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:35:01.828744 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.828670    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce66e16f-ca02-4e5b-af02-7034919579fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce66e16f-ca02-4e5b-af02-7034919579fe" (UID: "ce66e16f-ca02-4e5b-af02-7034919579fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:35:01.926757 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.926643    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-home\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:35:01.926757 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.926686    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-model-cache\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:35:01.926757 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.926705    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m96vs\" (UniqueName: \"kubernetes.io/projected/ce66e16f-ca02-4e5b-af02-7034919579fe-kube-api-access-m96vs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:35:01.926757 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.926719    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce66e16f-ca02-4e5b-af02-7034919579fe-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:35:01.926757 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:01.926733    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce66e16f-ca02-4e5b-af02-7034919579fe-dshm\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:35:02.627049 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:02.627012    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5" event={"ID":"ce66e16f-ca02-4e5b-af02-7034919579fe","Type":"ContainerDied","Data":"12fb4d91e7a9980dfe05472583969f5783d3f7a25367ee6b96679d5ec53ded30"}
Jul 27 17:35:02.627492 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:02.627062    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"
Jul 27 17:35:02.627492 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:02.627102    2566 scope.go:117] "RemoveContainer" containerID="ca4616245e086115fe5db063c95777890c006a01e56091472d8ce8fd40d1dcca"
Jul 27 17:35:02.643564 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:02.643538    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"]
Jul 27 17:35:02.646714 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:02.646691    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-j45h5"]
Jul 27 17:35:04.404036 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:04.403999    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce66e16f-ca02-4e5b-af02-7034919579fe" path="/var/lib/kubelet/pods/ce66e16f-ca02-4e5b-af02-7034919579fe/volumes"
Jul 27 17:35:40.688041 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.688003    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"]
Jul 27 17:35:40.690109 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.688418    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce66e16f-ca02-4e5b-af02-7034919579fe" containerName="main"
Jul 27 17:35:40.690109 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.688430    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce66e16f-ca02-4e5b-af02-7034919579fe" containerName="main"
Jul 27 17:35:40.690109 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.688500    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce66e16f-ca02-4e5b-af02-7034919579fe" containerName="main"
Jul 27 17:35:40.691418 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.691402    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:40.694310 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.694285    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 27 17:35:40.695352 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.695323    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 27 17:35:40.695471 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.695359    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-cstzn\""
Jul 27 17:35:40.695471 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.695396    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 27 17:35:40.699561 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.699535    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"]
Jul 27 17:35:40.775719 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.775687    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:40.775868 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.775729    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:40.775868 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.775807    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:40.775868 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.775846    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xsz7\" (UniqueName: \"kubernetes.io/projected/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-kube-api-access-8xsz7\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:40.776027 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.775887    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:40.776027 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.775986    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:40.876945 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.876909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:40.877114 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.876962    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:40.877114 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.876981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xsz7\" (UniqueName: \"kubernetes.io/projected/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-kube-api-access-8xsz7\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:40.877114 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.877016    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:40.877114 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.877044    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:40.877114 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.877086    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:40.877424 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.877388    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:40.877556 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.877416    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:40.877556 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.877500    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:40.879289 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.879267    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:40.879577 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.879561    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:40.885146 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:40.885121    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xsz7\" (UniqueName: \"kubernetes.io/projected/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-kube-api-access-8xsz7\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:41.002458 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:41.002427    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:41.136467 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:41.136307    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"]
Jul 27 17:35:41.139776 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:35:41.139750    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45bc8eb_76dc_4080_b313_fe5fed7d2c83.slice/crio-38dff71d4e17227858e680e721d54694b469774361147b2b297ecccab9a7da6f WatchSource:0}: Error finding container 38dff71d4e17227858e680e721d54694b469774361147b2b297ecccab9a7da6f: Status 404 returned error can't find the container with id 38dff71d4e17227858e680e721d54694b469774361147b2b297ecccab9a7da6f
Jul 27 17:35:41.840322 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:41.840286    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx" event={"ID":"b45bc8eb-76dc-4080-b313-fe5fed7d2c83","Type":"ContainerStarted","Data":"752669f1ae53b0feaa458b8f860dae9bea906671bb12c2107ddb06dcbc219d16"}
Jul 27 17:35:41.840698 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:41.840346    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx" event={"ID":"b45bc8eb-76dc-4080-b313-fe5fed7d2c83","Type":"ContainerStarted","Data":"38dff71d4e17227858e680e721d54694b469774361147b2b297ecccab9a7da6f"}
Jul 27 17:35:41.857966 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:41.857860    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx" podStartSLOduration=1.857845738 podStartE2EDuration="1.857845738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:35:41.856523277 +0000 UTC m=+2085.931787763" watchObservedRunningTime="2026-07-27 17:35:41.857845738 +0000 UTC m=+2085.933110259"
Jul 27 17:35:42.003370 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:42.003336    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:51.003214 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:51.003161    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:51.016388 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:51.016360    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:51.921301 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:51.921268    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:35:57.503292 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:57.503260    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:35:57.512090 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:35:57.512064    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:37:18.130304 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.130263    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"]
Jul 27 17:37:18.140230 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:37:18.140188    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:37:18.140230 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:37:18.140230 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:37:18.140230 ip-10-0-143-86 kubenswrapper[2566]:  > containerID="752669f1ae53b0feaa458b8f860dae9bea906671bb12c2107ddb06dcbc219d16" cmd=["/bin/sleep","15"]
Jul 27 17:37:18.140580 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.140251    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx" message="" err=<
Jul 27 17:37:18.140580 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:37:18.140580 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:37:18.140580 ip-10-0-143-86 kubenswrapper[2566]:  >
Jul 27 17:37:18.140580 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:37:18.140268    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:37:18.140580 ip-10-0-143-86 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:37:18.140580 ip-10-0-143-86 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Jul 27 17:37:18.140580 ip-10-0-143-86 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx" podUID="b45bc8eb-76dc-4080-b313-fe5fed7d2c83" containerName="main" containerID="cri-o://752669f1ae53b0feaa458b8f860dae9bea906671bb12c2107ddb06dcbc219d16"
Jul 27 17:37:18.140580 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.140317    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx" podUID="b45bc8eb-76dc-4080-b313-fe5fed7d2c83" containerName="main" containerID="cri-o://752669f1ae53b0feaa458b8f860dae9bea906671bb12c2107ddb06dcbc219d16" gracePeriod=60
Jul 27 17:37:18.390253 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.390141    2566 generic.go:356] "Generic (PLEG): container finished" podID="b45bc8eb-76dc-4080-b313-fe5fed7d2c83" containerID="752669f1ae53b0feaa458b8f860dae9bea906671bb12c2107ddb06dcbc219d16" exitCode=0
Jul 27 17:37:18.390394 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.390300    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx" event={"ID":"b45bc8eb-76dc-4080-b313-fe5fed7d2c83","Type":"ContainerDied","Data":"752669f1ae53b0feaa458b8f860dae9bea906671bb12c2107ddb06dcbc219d16"}
Jul 27 17:37:18.491479 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.491454    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:37:18.560844 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.560810    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-model-cache\") pod \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") "
Jul 27 17:37:18.561016 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.560856    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-tmp-dir\") pod \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") "
Jul 27 17:37:18.561016 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.560881    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-dshm\") pod \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") "
Jul 27 17:37:18.561016 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.560911    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xsz7\" (UniqueName: \"kubernetes.io/projected/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-kube-api-access-8xsz7\") pod \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") "
Jul 27 17:37:18.561016 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.560943    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-home\") pod \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") "
Jul 27 17:37:18.561016 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.560980    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-tls-certs\") pod \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\" (UID: \"b45bc8eb-76dc-4080-b313-fe5fed7d2c83\") "
Jul 27 17:37:18.561336 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.561116    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-model-cache" (OuterVolumeSpecName: "model-cache") pod "b45bc8eb-76dc-4080-b313-fe5fed7d2c83" (UID: "b45bc8eb-76dc-4080-b313-fe5fed7d2c83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:37:18.561336 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.561212    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b45bc8eb-76dc-4080-b313-fe5fed7d2c83" (UID: "b45bc8eb-76dc-4080-b313-fe5fed7d2c83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:37:18.561336 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.561231    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-home" (OuterVolumeSpecName: "home") pod "b45bc8eb-76dc-4080-b313-fe5fed7d2c83" (UID: "b45bc8eb-76dc-4080-b313-fe5fed7d2c83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:37:18.561336 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.561316    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-home\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:37:18.561336 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.561333    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-model-cache\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:37:18.561517 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.561349    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-tmp-dir\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:37:18.563154 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.563133    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-dshm" (OuterVolumeSpecName: "dshm") pod "b45bc8eb-76dc-4080-b313-fe5fed7d2c83" (UID: "b45bc8eb-76dc-4080-b313-fe5fed7d2c83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:37:18.563238 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.563158    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b45bc8eb-76dc-4080-b313-fe5fed7d2c83" (UID: "b45bc8eb-76dc-4080-b313-fe5fed7d2c83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:37:18.563329 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.563308    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-kube-api-access-8xsz7" (OuterVolumeSpecName: "kube-api-access-8xsz7") pod "b45bc8eb-76dc-4080-b313-fe5fed7d2c83" (UID: "b45bc8eb-76dc-4080-b313-fe5fed7d2c83"). InnerVolumeSpecName "kube-api-access-8xsz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:37:18.662005 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.661911    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-dshm\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:37:18.662005 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.661946    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xsz7\" (UniqueName: \"kubernetes.io/projected/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-kube-api-access-8xsz7\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:37:18.662005 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:18.661960    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b45bc8eb-76dc-4080-b313-fe5fed7d2c83-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:37:19.397137 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:19.397109    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"
Jul 27 17:37:19.397557 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:19.397140    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx" event={"ID":"b45bc8eb-76dc-4080-b313-fe5fed7d2c83","Type":"ContainerDied","Data":"38dff71d4e17227858e680e721d54694b469774361147b2b297ecccab9a7da6f"}
Jul 27 17:37:19.397557 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:19.397224    2566 scope.go:117] "RemoveContainer" containerID="752669f1ae53b0feaa458b8f860dae9bea906671bb12c2107ddb06dcbc219d16"
Jul 27 17:37:19.419375 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:19.419350    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"]
Jul 27 17:37:19.422805 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:19.422783    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rlqlx"]
Jul 27 17:37:20.403646 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:20.403612    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b45bc8eb-76dc-4080-b313-fe5fed7d2c83" path="/var/lib/kubelet/pods/b45bc8eb-76dc-4080-b313-fe5fed7d2c83/volumes"
Jul 27 17:37:41.401485 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.401449    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"]
Jul 27 17:37:41.401947 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.401822    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b45bc8eb-76dc-4080-b313-fe5fed7d2c83" containerName="main"
Jul 27 17:37:41.401947 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.401833    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b45bc8eb-76dc-4080-b313-fe5fed7d2c83" containerName="main"
Jul 27 17:37:41.401947 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.401909    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b45bc8eb-76dc-4080-b313-fe5fed7d2c83" containerName="main"
Jul 27 17:37:41.405003 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.404984    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.407538 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.407512    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Jul 27 17:37:41.407659 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.407610    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Jul 27 17:37:41.407730 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.407711    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 17:37:41.407782 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.407712    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-n2xlj\""
Jul 27 17:37:41.407857 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.407839    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 27 17:37:41.408365 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.408351    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Jul 27 17:37:41.415311 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.415288    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"]
Jul 27 17:37:41.477344 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.477314    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67qzp\" (UniqueName: \"kubernetes.io/projected/76c2bc41-f474-4c2f-a718-44f0f04753a0-kube-api-access-67qzp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.477517 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.477361    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76c2bc41-f474-4c2f-a718-44f0f04753a0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.477517 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.477381    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76c2bc41-f474-4c2f-a718-44f0f04753a0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.477517 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.477466    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.477517 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.477502    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.477731 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.477522    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.477731 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.477554    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.477731 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.477653    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.578101 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.578060    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.578101 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.578106    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.578362 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.578126    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.578362 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.578155    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.578362 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.578210    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.578362 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.578251    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67qzp\" (UniqueName: \"kubernetes.io/projected/76c2bc41-f474-4c2f-a718-44f0f04753a0-kube-api-access-67qzp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.578362 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.578278    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76c2bc41-f474-4c2f-a718-44f0f04753a0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.578362 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.578295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76c2bc41-f474-4c2f-a718-44f0f04753a0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.578681 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.578564    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.578681 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.578601    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.578828 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.578808    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.578886 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.578840    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.579120 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.579103    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76c2bc41-f474-4c2f-a718-44f0f04753a0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.580531 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.580513    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.580995 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.580976    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76c2bc41-f474-4c2f-a718-44f0f04753a0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.585961 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.585940    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67qzp\" (UniqueName: \"kubernetes.io/projected/76c2bc41-f474-4c2f-a718-44f0f04753a0-kube-api-access-67qzp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.716037 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.715989    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:41.860973 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:41.860937    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"]
Jul 27 17:37:41.862419 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:37:41.862386    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c2bc41_f474_4c2f_a718_44f0f04753a0.slice/crio-a9ac09d3ef6fc28aa96a56cc069ddee1c6c03a1135882077d12f98213922e589 WatchSource:0}: Error finding container a9ac09d3ef6fc28aa96a56cc069ddee1c6c03a1135882077d12f98213922e589: Status 404 returned error can't find the container with id a9ac09d3ef6fc28aa96a56cc069ddee1c6c03a1135882077d12f98213922e589
Jul 27 17:37:42.523297 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:42.523247    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw" event={"ID":"76c2bc41-f474-4c2f-a718-44f0f04753a0","Type":"ContainerStarted","Data":"7818393f8e676d5de8652ff5624f9d81b5926e88af34f14efae810be2c27e08d"}
Jul 27 17:37:42.523764 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:42.523326    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw" event={"ID":"76c2bc41-f474-4c2f-a718-44f0f04753a0","Type":"ContainerStarted","Data":"a9ac09d3ef6fc28aa96a56cc069ddee1c6c03a1135882077d12f98213922e589"}
Jul 27 17:37:50.571121 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:50.571084    2566 generic.go:356] "Generic (PLEG): container finished" podID="76c2bc41-f474-4c2f-a718-44f0f04753a0" containerID="7818393f8e676d5de8652ff5624f9d81b5926e88af34f14efae810be2c27e08d" exitCode=0
Jul 27 17:37:50.571574 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:50.571156    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw" event={"ID":"76c2bc41-f474-4c2f-a718-44f0f04753a0","Type":"ContainerDied","Data":"7818393f8e676d5de8652ff5624f9d81b5926e88af34f14efae810be2c27e08d"}
Jul 27 17:37:51.579897 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:51.579853    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw" event={"ID":"76c2bc41-f474-4c2f-a718-44f0f04753a0","Type":"ContainerStarted","Data":"9b3db4732c173fa02f950fddbf6acaf85a160add1838c7c7aece6803960fbadf"}
Jul 27 17:37:51.603611 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:51.603540    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw" podStartSLOduration=2.391059232 podStartE2EDuration="10.603518788s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.212459556s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:37:51.600615647 +0000 UTC m=+2215.675880134" watchObservedRunningTime="2026-07-27 17:37:51.603518788 +0000 UTC m=+2215.678783276"
Jul 27 17:37:51.716648 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:51.716598    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:51.716836 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:51.716679    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:37:51.718487 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:37:51.718454    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw" podUID="76c2bc41-f474-4c2f-a718-44f0f04753a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 17:38:01.717254 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:38:01.717207    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw" podUID="76c2bc41-f474-4c2f-a718-44f0f04753a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 17:38:11.716946 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:38:11.716893    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw" podUID="76c2bc41-f474-4c2f-a718-44f0f04753a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 17:38:21.716755 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:38:21.716656    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw" podUID="76c2bc41-f474-4c2f-a718-44f0f04753a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 17:38:31.716871 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:38:31.716816    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw" podUID="76c2bc41-f474-4c2f-a718-44f0f04753a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 17:38:41.716848 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:38:41.716801    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw" podUID="76c2bc41-f474-4c2f-a718-44f0f04753a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 17:38:51.717359 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:38:51.717311    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw" podUID="76c2bc41-f474-4c2f-a718-44f0f04753a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 17:39:01.716717 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:01.716664    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw" podUID="76c2bc41-f474-4c2f-a718-44f0f04753a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 17:39:11.726428 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:11.726391    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:39:11.733961 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:11.733939    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:39:37.625193 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:37.625139    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"]
Jul 27 17:39:52.648263 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:52.648196    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw" podUID="76c2bc41-f474-4c2f-a718-44f0f04753a0" containerName="main" containerID="cri-o://9b3db4732c173fa02f950fddbf6acaf85a160add1838c7c7aece6803960fbadf" gracePeriod=45
Jul 27 17:39:52.727877 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:52.727819    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw" podUID="76c2bc41-f474-4c2f-a718-44f0f04753a0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 17:39:53.301544 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.301518    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:39:53.344315 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.344227    2566 generic.go:356] "Generic (PLEG): container finished" podID="76c2bc41-f474-4c2f-a718-44f0f04753a0" containerID="9b3db4732c173fa02f950fddbf6acaf85a160add1838c7c7aece6803960fbadf" exitCode=0
Jul 27 17:39:53.344315 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.344276    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"
Jul 27 17:39:53.344506 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.344321    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw" event={"ID":"76c2bc41-f474-4c2f-a718-44f0f04753a0","Type":"ContainerDied","Data":"9b3db4732c173fa02f950fddbf6acaf85a160add1838c7c7aece6803960fbadf"}
Jul 27 17:39:53.344506 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.344391    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw" event={"ID":"76c2bc41-f474-4c2f-a718-44f0f04753a0","Type":"ContainerDied","Data":"a9ac09d3ef6fc28aa96a56cc069ddee1c6c03a1135882077d12f98213922e589"}
Jul 27 17:39:53.344506 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.344426    2566 scope.go:117] "RemoveContainer" containerID="9b3db4732c173fa02f950fddbf6acaf85a160add1838c7c7aece6803960fbadf"
Jul 27 17:39:53.353626 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.353608    2566 scope.go:117] "RemoveContainer" containerID="7818393f8e676d5de8652ff5624f9d81b5926e88af34f14efae810be2c27e08d"
Jul 27 17:39:53.362536 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.362518    2566 scope.go:117] "RemoveContainer" containerID="9b3db4732c173fa02f950fddbf6acaf85a160add1838c7c7aece6803960fbadf"
Jul 27 17:39:53.362807 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:39:53.362780    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3db4732c173fa02f950fddbf6acaf85a160add1838c7c7aece6803960fbadf\": container with ID starting with 9b3db4732c173fa02f950fddbf6acaf85a160add1838c7c7aece6803960fbadf not found: ID does not exist" containerID="9b3db4732c173fa02f950fddbf6acaf85a160add1838c7c7aece6803960fbadf"
Jul 27 17:39:53.362910 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.362815    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3db4732c173fa02f950fddbf6acaf85a160add1838c7c7aece6803960fbadf"} err="failed to get container status \"9b3db4732c173fa02f950fddbf6acaf85a160add1838c7c7aece6803960fbadf\": rpc error: code = NotFound desc = could not find container \"9b3db4732c173fa02f950fddbf6acaf85a160add1838c7c7aece6803960fbadf\": container with ID starting with 9b3db4732c173fa02f950fddbf6acaf85a160add1838c7c7aece6803960fbadf not found: ID does not exist"
Jul 27 17:39:53.362910 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.362833    2566 scope.go:117] "RemoveContainer" containerID="7818393f8e676d5de8652ff5624f9d81b5926e88af34f14efae810be2c27e08d"
Jul 27 17:39:53.363071 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:39:53.363051    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7818393f8e676d5de8652ff5624f9d81b5926e88af34f14efae810be2c27e08d\": container with ID starting with 7818393f8e676d5de8652ff5624f9d81b5926e88af34f14efae810be2c27e08d not found: ID does not exist" containerID="7818393f8e676d5de8652ff5624f9d81b5926e88af34f14efae810be2c27e08d"
Jul 27 17:39:53.363112 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.363078    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7818393f8e676d5de8652ff5624f9d81b5926e88af34f14efae810be2c27e08d"} err="failed to get container status \"7818393f8e676d5de8652ff5624f9d81b5926e88af34f14efae810be2c27e08d\": rpc error: code = NotFound desc = could not find container \"7818393f8e676d5de8652ff5624f9d81b5926e88af34f14efae810be2c27e08d\": container with ID starting with 7818393f8e676d5de8652ff5624f9d81b5926e88af34f14efae810be2c27e08d not found: ID does not exist"
Jul 27 17:39:53.436553 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.436511    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-dshm\") pod \"76c2bc41-f474-4c2f-a718-44f0f04753a0\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") "
Jul 27 17:39:53.436553 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.436552    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-tmp-dir\") pod \"76c2bc41-f474-4c2f-a718-44f0f04753a0\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") "
Jul 27 17:39:53.436806 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.436597    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76c2bc41-f474-4c2f-a718-44f0f04753a0-tls-certs\") pod \"76c2bc41-f474-4c2f-a718-44f0f04753a0\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") "
Jul 27 17:39:53.436806 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.436630    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-kserve-provision-location\") pod \"76c2bc41-f474-4c2f-a718-44f0f04753a0\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") "
Jul 27 17:39:53.436806 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.436666    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76c2bc41-f474-4c2f-a718-44f0f04753a0-cabundle-cert\") pod \"76c2bc41-f474-4c2f-a718-44f0f04753a0\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") "
Jul 27 17:39:53.436806 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.436732    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-model-cache\") pod \"76c2bc41-f474-4c2f-a718-44f0f04753a0\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") "
Jul 27 17:39:53.436806 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.436760    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67qzp\" (UniqueName: \"kubernetes.io/projected/76c2bc41-f474-4c2f-a718-44f0f04753a0-kube-api-access-67qzp\") pod \"76c2bc41-f474-4c2f-a718-44f0f04753a0\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") "
Jul 27 17:39:53.436806 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.436783    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-home\") pod \"76c2bc41-f474-4c2f-a718-44f0f04753a0\" (UID: \"76c2bc41-f474-4c2f-a718-44f0f04753a0\") "
Jul 27 17:39:53.437132 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.437013    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "76c2bc41-f474-4c2f-a718-44f0f04753a0" (UID: "76c2bc41-f474-4c2f-a718-44f0f04753a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:39:53.437132 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.437053    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c2bc41-f474-4c2f-a718-44f0f04753a0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "76c2bc41-f474-4c2f-a718-44f0f04753a0" (UID: "76c2bc41-f474-4c2f-a718-44f0f04753a0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:39:53.437435 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.437407    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-home" (OuterVolumeSpecName: "home") pod "76c2bc41-f474-4c2f-a718-44f0f04753a0" (UID: "76c2bc41-f474-4c2f-a718-44f0f04753a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:39:53.439007 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.438980    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-dshm" (OuterVolumeSpecName: "dshm") pod "76c2bc41-f474-4c2f-a718-44f0f04753a0" (UID: "76c2bc41-f474-4c2f-a718-44f0f04753a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:39:53.439318 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.439290    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c2bc41-f474-4c2f-a718-44f0f04753a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76c2bc41-f474-4c2f-a718-44f0f04753a0" (UID: "76c2bc41-f474-4c2f-a718-44f0f04753a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:39:53.439421 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.439319    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c2bc41-f474-4c2f-a718-44f0f04753a0-kube-api-access-67qzp" (OuterVolumeSpecName: "kube-api-access-67qzp") pod "76c2bc41-f474-4c2f-a718-44f0f04753a0" (UID: "76c2bc41-f474-4c2f-a718-44f0f04753a0"). InnerVolumeSpecName "kube-api-access-67qzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:39:53.447862 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.447840    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76c2bc41-f474-4c2f-a718-44f0f04753a0" (UID: "76c2bc41-f474-4c2f-a718-44f0f04753a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:39:53.506066 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.506025    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "76c2bc41-f474-4c2f-a718-44f0f04753a0" (UID: "76c2bc41-f474-4c2f-a718-44f0f04753a0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:39:53.537943 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.537909    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76c2bc41-f474-4c2f-a718-44f0f04753a0-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:39:53.537943 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.537942    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-kserve-provision-location\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:39:53.538097 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.537954    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76c2bc41-f474-4c2f-a718-44f0f04753a0-cabundle-cert\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:39:53.538097 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.537964    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-model-cache\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:39:53.538097 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.537974    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67qzp\" (UniqueName: \"kubernetes.io/projected/76c2bc41-f474-4c2f-a718-44f0f04753a0-kube-api-access-67qzp\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:39:53.538097 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.537982    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-home\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:39:53.538097 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.537989    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-dshm\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:39:53.538097 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.537998    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c2bc41-f474-4c2f-a718-44f0f04753a0-tmp-dir\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:39:53.668001 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.667967    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"]
Jul 27 17:39:53.671721 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:53.671696    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-74cff8cc8b-4xjhw"]
Jul 27 17:39:54.403361 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:39:54.403329    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76c2bc41-f474-4c2f-a718-44f0f04753a0" path="/var/lib/kubelet/pods/76c2bc41-f474-4c2f-a718-44f0f04753a0/volumes"
Jul 27 17:40:05.007181 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.007127    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"]
Jul 27 17:40:05.007642 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.007616    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c2bc41-f474-4c2f-a718-44f0f04753a0" containerName="main"
Jul 27 17:40:05.007642 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.007634    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c2bc41-f474-4c2f-a718-44f0f04753a0" containerName="main"
Jul 27 17:40:05.007731 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.007647    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c2bc41-f474-4c2f-a718-44f0f04753a0" containerName="storage-initializer"
Jul 27 17:40:05.007731 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.007653    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c2bc41-f474-4c2f-a718-44f0f04753a0" containerName="storage-initializer"
Jul 27 17:40:05.007731 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.007725    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="76c2bc41-f474-4c2f-a718-44f0f04753a0" containerName="main"
Jul 27 17:40:05.010955 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.010936    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.013708 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.013677    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Jul 27 17:40:05.013835 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.013707    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 17:40:05.013923 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.013851    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Jul 27 17:40:05.013923 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.013885    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-pbvkd\""
Jul 27 17:40:05.014034 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.014009    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Jul 27 17:40:05.014187 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.014156    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 27 17:40:05.023738 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.023713    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"]
Jul 27 17:40:05.041633 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.041601    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.041782 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.041638    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.041782 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.041670    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.041782 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.041719    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb9bcff-9448-483e-8cd5-87145bfafb7c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.041782 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.041741    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bb9bcff-9448-483e-8cd5-87145bfafb7c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.041950 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.041827    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.041950 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.041865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.041950 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.041917    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bsk4\" (UniqueName: \"kubernetes.io/projected/4bb9bcff-9448-483e-8cd5-87145bfafb7c-kube-api-access-2bsk4\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.142579 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.142536    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bsk4\" (UniqueName: \"kubernetes.io/projected/4bb9bcff-9448-483e-8cd5-87145bfafb7c-kube-api-access-2bsk4\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.142803 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.142600    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.142803 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.142634    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.142803 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.142676    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.142803 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.142705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb9bcff-9448-483e-8cd5-87145bfafb7c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.142803 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.142727    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bb9bcff-9448-483e-8cd5-87145bfafb7c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.142803 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.142780    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.143138 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.142803    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.143249 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.143225    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.143322 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.143289    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.143380 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.143336    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.143380 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.143355    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.143795 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.143775    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bb9bcff-9448-483e-8cd5-87145bfafb7c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.145759 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.145734    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.146021 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.146005    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb9bcff-9448-483e-8cd5-87145bfafb7c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.151731 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.151709    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bsk4\" (UniqueName: \"kubernetes.io/projected/4bb9bcff-9448-483e-8cd5-87145bfafb7c-kube-api-access-2bsk4\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.321756 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.321673    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:05.468989 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:05.468945    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"]
Jul 27 17:40:05.477255 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:40:05.477223    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb9bcff_9448_483e_8cd5_87145bfafb7c.slice/crio-f17fcc2067740e34de3fb47efdfb8ed10e341f503ea82df9f105c385be674ccb WatchSource:0}: Error finding container f17fcc2067740e34de3fb47efdfb8ed10e341f503ea82df9f105c385be674ccb: Status 404 returned error can't find the container with id f17fcc2067740e34de3fb47efdfb8ed10e341f503ea82df9f105c385be674ccb
Jul 27 17:40:06.417472 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:06.417430    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln" event={"ID":"4bb9bcff-9448-483e-8cd5-87145bfafb7c","Type":"ContainerStarted","Data":"60f58fab5bd8a45f09ba298762c980dfe1ab0d41a59e7b84b1d8c8f5626673fb"}
Jul 27 17:40:06.417943 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:06.417501    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln" event={"ID":"4bb9bcff-9448-483e-8cd5-87145bfafb7c","Type":"ContainerStarted","Data":"f17fcc2067740e34de3fb47efdfb8ed10e341f503ea82df9f105c385be674ccb"}
Jul 27 17:40:15.468788 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:15.468751    2566 generic.go:356] "Generic (PLEG): container finished" podID="4bb9bcff-9448-483e-8cd5-87145bfafb7c" containerID="60f58fab5bd8a45f09ba298762c980dfe1ab0d41a59e7b84b1d8c8f5626673fb" exitCode=0
Jul 27 17:40:15.469182 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:15.468826    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln" event={"ID":"4bb9bcff-9448-483e-8cd5-87145bfafb7c","Type":"ContainerDied","Data":"60f58fab5bd8a45f09ba298762c980dfe1ab0d41a59e7b84b1d8c8f5626673fb"}
Jul 27 17:40:16.478104 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:16.478065    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln" event={"ID":"4bb9bcff-9448-483e-8cd5-87145bfafb7c","Type":"ContainerStarted","Data":"39aa417bc032cab99750452aafda885f726f95e7d1cdbcf9c24b8d7679169247"}
Jul 27 17:40:16.515053 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:16.514979    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln" podStartSLOduration=3.552625164 podStartE2EDuration="12.514957134s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.96233197s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:40:16.512112954 +0000 UTC m=+2360.587377455" watchObservedRunningTime="2026-07-27 17:40:16.514957134 +0000 UTC m=+2360.590221623"
Jul 27 17:40:17.322187 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:17.322125    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:25.322416 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:25.322378    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:40:25.323993 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:25.323961    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln" podUID="4bb9bcff-9448-483e-8cd5-87145bfafb7c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 27 17:40:35.322356 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:35.322312    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln" podUID="4bb9bcff-9448-483e-8cd5-87145bfafb7c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 27 17:40:45.323789 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:45.323737    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln" podUID="4bb9bcff-9448-483e-8cd5-87145bfafb7c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 27 17:40:55.322719 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:55.322675    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln" podUID="4bb9bcff-9448-483e-8cd5-87145bfafb7c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 27 17:40:57.551328 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:57.551285    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:40:57.560732 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:40:57.560709    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:41:05.322506 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:41:05.322459    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln" podUID="4bb9bcff-9448-483e-8cd5-87145bfafb7c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 27 17:41:15.322369 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:41:15.322317    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln" podUID="4bb9bcff-9448-483e-8cd5-87145bfafb7c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 27 17:41:25.322414 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:41:25.322316    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln" podUID="4bb9bcff-9448-483e-8cd5-87145bfafb7c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 27 17:41:35.332944 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:41:35.332906    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:41:35.341096 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:41:35.341065    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:41:55.396598 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:41:55.396562    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"]
Jul 27 17:42:10.418611 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:10.418553    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln" podUID="4bb9bcff-9448-483e-8cd5-87145bfafb7c" containerName="main" containerID="cri-o://39aa417bc032cab99750452aafda885f726f95e7d1cdbcf9c24b8d7679169247" gracePeriod=45
Jul 27 17:42:10.975981 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:10.975955    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:42:11.041300 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.041226    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bb9bcff-9448-483e-8cd5-87145bfafb7c-cabundle-cert\") pod \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") "
Jul 27 17:42:11.041300 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.041270    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-tmp-dir\") pod \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") "
Jul 27 17:42:11.041300 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.041295    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bsk4\" (UniqueName: \"kubernetes.io/projected/4bb9bcff-9448-483e-8cd5-87145bfafb7c-kube-api-access-2bsk4\") pod \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") "
Jul 27 17:42:11.041563 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.041416    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-model-cache\") pod \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") "
Jul 27 17:42:11.041563 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.041464    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb9bcff-9448-483e-8cd5-87145bfafb7c-tls-certs\") pod \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") "
Jul 27 17:42:11.041563 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.041494    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-dshm\") pod \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") "
Jul 27 17:42:11.041563 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.041557    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-kserve-provision-location\") pod \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") "
Jul 27 17:42:11.041773 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.041555    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4bb9bcff-9448-483e-8cd5-87145bfafb7c" (UID: "4bb9bcff-9448-483e-8cd5-87145bfafb7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:42:11.041773 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.041691    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-home\") pod \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\" (UID: \"4bb9bcff-9448-483e-8cd5-87145bfafb7c\") "
Jul 27 17:42:11.041896 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.041685    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4bb9bcff-9448-483e-8cd5-87145bfafb7c" (UID: "4bb9bcff-9448-483e-8cd5-87145bfafb7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:42:11.041896 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.041718    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb9bcff-9448-483e-8cd5-87145bfafb7c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4bb9bcff-9448-483e-8cd5-87145bfafb7c" (UID: "4bb9bcff-9448-483e-8cd5-87145bfafb7c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:42:11.042031 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.042007    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-home" (OuterVolumeSpecName: "home") pod "4bb9bcff-9448-483e-8cd5-87145bfafb7c" (UID: "4bb9bcff-9448-483e-8cd5-87145bfafb7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:42:11.042031 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.042022    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bb9bcff-9448-483e-8cd5-87145bfafb7c-cabundle-cert\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:42:11.042208 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.042040    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-tmp-dir\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:42:11.042208 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.042053    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-model-cache\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:42:11.043656 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.043632    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb9bcff-9448-483e-8cd5-87145bfafb7c-kube-api-access-2bsk4" (OuterVolumeSpecName: "kube-api-access-2bsk4") pod "4bb9bcff-9448-483e-8cd5-87145bfafb7c" (UID: "4bb9bcff-9448-483e-8cd5-87145bfafb7c"). InnerVolumeSpecName "kube-api-access-2bsk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:42:11.044076 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.044055    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-dshm" (OuterVolumeSpecName: "dshm") pod "4bb9bcff-9448-483e-8cd5-87145bfafb7c" (UID: "4bb9bcff-9448-483e-8cd5-87145bfafb7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:42:11.044147 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.044065    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb9bcff-9448-483e-8cd5-87145bfafb7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4bb9bcff-9448-483e-8cd5-87145bfafb7c" (UID: "4bb9bcff-9448-483e-8cd5-87145bfafb7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:42:11.105912 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.105871    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4bb9bcff-9448-483e-8cd5-87145bfafb7c" (UID: "4bb9bcff-9448-483e-8cd5-87145bfafb7c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:42:11.142583 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.142553    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bsk4\" (UniqueName: \"kubernetes.io/projected/4bb9bcff-9448-483e-8cd5-87145bfafb7c-kube-api-access-2bsk4\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:42:11.142583 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.142581    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb9bcff-9448-483e-8cd5-87145bfafb7c-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:42:11.142775 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.142593    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-dshm\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:42:11.142775 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.142603    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-kserve-provision-location\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:42:11.142775 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.142612    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb9bcff-9448-483e-8cd5-87145bfafb7c-home\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:42:11.194486 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.194450    2566 generic.go:356] "Generic (PLEG): container finished" podID="4bb9bcff-9448-483e-8cd5-87145bfafb7c" containerID="39aa417bc032cab99750452aafda885f726f95e7d1cdbcf9c24b8d7679169247" exitCode=0
Jul 27 17:42:11.194648 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.194501    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"
Jul 27 17:42:11.194648 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.194532    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln" event={"ID":"4bb9bcff-9448-483e-8cd5-87145bfafb7c","Type":"ContainerDied","Data":"39aa417bc032cab99750452aafda885f726f95e7d1cdbcf9c24b8d7679169247"}
Jul 27 17:42:11.194648 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.194613    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln" event={"ID":"4bb9bcff-9448-483e-8cd5-87145bfafb7c","Type":"ContainerDied","Data":"f17fcc2067740e34de3fb47efdfb8ed10e341f503ea82df9f105c385be674ccb"}
Jul 27 17:42:11.194648 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.194645    2566 scope.go:117] "RemoveContainer" containerID="39aa417bc032cab99750452aafda885f726f95e7d1cdbcf9c24b8d7679169247"
Jul 27 17:42:11.204349 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.204322    2566 scope.go:117] "RemoveContainer" containerID="60f58fab5bd8a45f09ba298762c980dfe1ab0d41a59e7b84b1d8c8f5626673fb"
Jul 27 17:42:11.212515 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.212499    2566 scope.go:117] "RemoveContainer" containerID="39aa417bc032cab99750452aafda885f726f95e7d1cdbcf9c24b8d7679169247"
Jul 27 17:42:11.212762 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:42:11.212743    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39aa417bc032cab99750452aafda885f726f95e7d1cdbcf9c24b8d7679169247\": container with ID starting with 39aa417bc032cab99750452aafda885f726f95e7d1cdbcf9c24b8d7679169247 not found: ID does not exist" containerID="39aa417bc032cab99750452aafda885f726f95e7d1cdbcf9c24b8d7679169247"
Jul 27 17:42:11.212805 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.212771    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39aa417bc032cab99750452aafda885f726f95e7d1cdbcf9c24b8d7679169247"} err="failed to get container status \"39aa417bc032cab99750452aafda885f726f95e7d1cdbcf9c24b8d7679169247\": rpc error: code = NotFound desc = could not find container \"39aa417bc032cab99750452aafda885f726f95e7d1cdbcf9c24b8d7679169247\": container with ID starting with 39aa417bc032cab99750452aafda885f726f95e7d1cdbcf9c24b8d7679169247 not found: ID does not exist"
Jul 27 17:42:11.212805 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.212788    2566 scope.go:117] "RemoveContainer" containerID="60f58fab5bd8a45f09ba298762c980dfe1ab0d41a59e7b84b1d8c8f5626673fb"
Jul 27 17:42:11.213011 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:42:11.212994    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f58fab5bd8a45f09ba298762c980dfe1ab0d41a59e7b84b1d8c8f5626673fb\": container with ID starting with 60f58fab5bd8a45f09ba298762c980dfe1ab0d41a59e7b84b1d8c8f5626673fb not found: ID does not exist" containerID="60f58fab5bd8a45f09ba298762c980dfe1ab0d41a59e7b84b1d8c8f5626673fb"
Jul 27 17:42:11.213051 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.213016    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f58fab5bd8a45f09ba298762c980dfe1ab0d41a59e7b84b1d8c8f5626673fb"} err="failed to get container status \"60f58fab5bd8a45f09ba298762c980dfe1ab0d41a59e7b84b1d8c8f5626673fb\": rpc error: code = NotFound desc = could not find container \"60f58fab5bd8a45f09ba298762c980dfe1ab0d41a59e7b84b1d8c8f5626673fb\": container with ID starting with 60f58fab5bd8a45f09ba298762c980dfe1ab0d41a59e7b84b1d8c8f5626673fb not found: ID does not exist"
Jul 27 17:42:11.216911 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.216890    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"]
Jul 27 17:42:11.221245 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:11.221224    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-59ff99fc55-kcsln"]
Jul 27 17:42:12.403937 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:12.403900    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bb9bcff-9448-483e-8cd5-87145bfafb7c" path="/var/lib/kubelet/pods/4bb9bcff-9448-483e-8cd5-87145bfafb7c/volumes"
Jul 27 17:42:18.995006 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:18.994974    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq"]
Jul 27 17:42:18.995485 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:18.995436    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bb9bcff-9448-483e-8cd5-87145bfafb7c" containerName="main"
Jul 27 17:42:18.995485 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:18.995450    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bb9bcff-9448-483e-8cd5-87145bfafb7c" containerName="main"
Jul 27 17:42:18.995485 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:18.995460    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bb9bcff-9448-483e-8cd5-87145bfafb7c" containerName="storage-initializer"
Jul 27 17:42:18.995485 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:18.995466    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bb9bcff-9448-483e-8cd5-87145bfafb7c" containerName="storage-initializer"
Jul 27 17:42:18.995621 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:18.995545    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bb9bcff-9448-483e-8cd5-87145bfafb7c" containerName="main"
Jul 27 17:42:19.000056 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:19.000037    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq"
Jul 27 17:42:19.002547 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:19.002526    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 27 17:42:19.002839 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:19.002821    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-v5mkm\""
Jul 27 17:42:19.003671 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:19.003650    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 27 17:42:19.006181 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:19.006146    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq"]
Jul 27 17:42:19.111121 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:19.111082    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mrb4\" (UniqueName: \"kubernetes.io/projected/ea1882ee-2f37-4838-ac7a-004f42e93893-kube-api-access-2mrb4\") pod \"e2e-pvc-model-download-hgsqq\" (UID: \"ea1882ee-2f37-4838-ac7a-004f42e93893\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq"
Jul 27 17:42:19.212501 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:19.212462    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mrb4\" (UniqueName: \"kubernetes.io/projected/ea1882ee-2f37-4838-ac7a-004f42e93893-kube-api-access-2mrb4\") pod \"e2e-pvc-model-download-hgsqq\" (UID: \"ea1882ee-2f37-4838-ac7a-004f42e93893\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq"
Jul 27 17:42:19.221093 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:19.221062    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mrb4\" (UniqueName: \"kubernetes.io/projected/ea1882ee-2f37-4838-ac7a-004f42e93893-kube-api-access-2mrb4\") pod \"e2e-pvc-model-download-hgsqq\" (UID: \"ea1882ee-2f37-4838-ac7a-004f42e93893\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq"
Jul 27 17:42:25.673746 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:25.673705    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") pod \"e2e-pvc-model-download-hgsqq\" (UID: \"ea1882ee-2f37-4838-ac7a-004f42e93893\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq"
Jul 27 17:42:25.679594 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:25.679569    2566 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") pod \"e2e-pvc-model-download-hgsqq\" (UID: \"ea1882ee-2f37-4838-ac7a-004f42e93893\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq"
Jul 27 17:42:25.775129 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:25.775093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") pod \"e2e-pvc-model-download-hgsqq\" (UID: \"ea1882ee-2f37-4838-ac7a-004f42e93893\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq"
Jul 27 17:42:25.775312 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:25.775247    2566 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") pod \"e2e-pvc-model-download-hgsqq\" (UID: \"ea1882ee-2f37-4838-ac7a-004f42e93893\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq"
Jul 27 17:42:25.778139 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:25.778117    2566 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") pod \"e2e-pvc-model-download-hgsqq\" (UID: \"ea1882ee-2f37-4838-ac7a-004f42e93893\") DevicePath \"csi-0e8a7ed6a116a367a232c2eceb4606b3bc7c853843206c719654385d26f5ce26\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq"
Jul 27 17:42:25.995205 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:25.995150    2566 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") pod \"e2e-pvc-model-download-hgsqq\" (UID: \"ea1882ee-2f37-4838-ac7a-004f42e93893\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/c95aa4f5833a65df4bc95280ad8355c13b05bf90eb1c331f50d3810708c6f2c6/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq"
Jul 27 17:42:26.008585 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:26.008557    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") pod \"e2e-pvc-model-download-hgsqq\" (UID: \"ea1882ee-2f37-4838-ac7a-004f42e93893\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq"
Jul 27 17:42:26.209779 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:26.209726    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq"
Jul 27 17:42:26.344706 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:26.344673    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq"]
Jul 27 17:42:26.346391 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:42:26.346366    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea1882ee_2f37_4838_ac7a_004f42e93893.slice/crio-4715eedea53e8b4ca0c90b0be90f0673a9ef32e1e326c18b4ff80bbbc127b0af WatchSource:0}: Error finding container 4715eedea53e8b4ca0c90b0be90f0673a9ef32e1e326c18b4ff80bbbc127b0af: Status 404 returned error can't find the container with id 4715eedea53e8b4ca0c90b0be90f0673a9ef32e1e326c18b4ff80bbbc127b0af
Jul 27 17:42:27.288790 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:27.288748    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq" event={"ID":"ea1882ee-2f37-4838-ac7a-004f42e93893","Type":"ContainerStarted","Data":"d06ccfa7c06e0186f7639bd29476456bf1da895e32c331f3f5470981601705ad"}
Jul 27 17:42:27.289265 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:27.288815    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq" event={"ID":"ea1882ee-2f37-4838-ac7a-004f42e93893","Type":"ContainerStarted","Data":"4715eedea53e8b4ca0c90b0be90f0673a9ef32e1e326c18b4ff80bbbc127b0af"}
Jul 27 17:42:27.305915 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:27.305862    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq" podStartSLOduration=12.305837417 podStartE2EDuration="12.305837417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:42:27.305093687 +0000 UTC m=+2491.380358186" watchObservedRunningTime="2026-07-27 17:42:27.305837417 +0000 UTC m=+2491.381101904"
Jul 27 17:42:35.335228 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:35.335191    2566 generic.go:356] "Generic (PLEG): container finished" podID="ea1882ee-2f37-4838-ac7a-004f42e93893" containerID="d06ccfa7c06e0186f7639bd29476456bf1da895e32c331f3f5470981601705ad" exitCode=0
Jul 27 17:42:35.335228 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:35.335230    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq" event={"ID":"ea1882ee-2f37-4838-ac7a-004f42e93893","Type":"ContainerDied","Data":"d06ccfa7c06e0186f7639bd29476456bf1da895e32c331f3f5470981601705ad"}
Jul 27 17:42:36.490323 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:36.490299    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq"
Jul 27 17:42:36.571192 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:36.571139    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mrb4\" (UniqueName: \"kubernetes.io/projected/ea1882ee-2f37-4838-ac7a-004f42e93893-kube-api-access-2mrb4\") pod \"ea1882ee-2f37-4838-ac7a-004f42e93893\" (UID: \"ea1882ee-2f37-4838-ac7a-004f42e93893\") "
Jul 27 17:42:36.571388 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:36.571357    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") pod \"ea1882ee-2f37-4838-ac7a-004f42e93893\" (UID: \"ea1882ee-2f37-4838-ac7a-004f42e93893\") "
Jul 27 17:42:36.573386 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:36.573360    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea1882ee-2f37-4838-ac7a-004f42e93893-kube-api-access-2mrb4" (OuterVolumeSpecName: "kube-api-access-2mrb4") pod "ea1882ee-2f37-4838-ac7a-004f42e93893" (UID: "ea1882ee-2f37-4838-ac7a-004f42e93893"). InnerVolumeSpecName "kube-api-access-2mrb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:42:36.575271 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:36.575245    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f" (OuterVolumeSpecName: "model-storage") pod "ea1882ee-2f37-4838-ac7a-004f42e93893" (UID: "ea1882ee-2f37-4838-ac7a-004f42e93893"). InnerVolumeSpecName "pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 27 17:42:36.672962 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:36.672870    2566 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") on node \"ip-10-0-143-86.ec2.internal\" "
Jul 27 17:42:36.672962 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:36.672904    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mrb4\" (UniqueName: \"kubernetes.io/projected/ea1882ee-2f37-4838-ac7a-004f42e93893-kube-api-access-2mrb4\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:42:36.788925 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:36.788901    2566 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f") on node "ip-10-0-143-86.ec2.internal"
Jul 27 17:42:36.875578 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:36.875546    2566 reconciler_common.go:299] "Volume detached for volume \"pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"csi-0e8a7ed6a116a367a232c2eceb4606b3bc7c853843206c719654385d26f5ce26\""
Jul 27 17:42:37.346664 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:37.346632    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq"
Jul 27 17:42:37.346839 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:37.346636    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq" event={"ID":"ea1882ee-2f37-4838-ac7a-004f42e93893","Type":"ContainerDied","Data":"4715eedea53e8b4ca0c90b0be90f0673a9ef32e1e326c18b4ff80bbbc127b0af"}
Jul 27 17:42:37.346839 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:37.346751    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4715eedea53e8b4ca0c90b0be90f0673a9ef32e1e326c18b4ff80bbbc127b0af"
Jul 27 17:42:57.093985 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.093895    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"]
Jul 27 17:42:57.094425 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.094405    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea1882ee-2f37-4838-ac7a-004f42e93893" containerName="storage-initializer"
Jul 27 17:42:57.094468 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.094426    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea1882ee-2f37-4838-ac7a-004f42e93893" containerName="storage-initializer"
Jul 27 17:42:57.094507 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.094500    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea1882ee-2f37-4838-ac7a-004f42e93893" containerName="storage-initializer"
Jul 27 17:42:57.098746 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.098729    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:42:57.101468 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.101446    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-v5mkm\""
Jul 27 17:42:57.101627 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.101555    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 27 17:42:57.101627 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.101553    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 27 17:42:57.101743 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.101628    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 27 17:42:57.108260 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.108238    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"]
Jul 27 17:42:57.256517 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.256470    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:42:57.256517 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.256518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:42:57.256779 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.256629    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:42:57.256779 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.256675    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sgf9\" (UniqueName: \"kubernetes.io/projected/48a46d84-608a-428e-b5e0-c8337fb18da0-kube-api-access-2sgf9\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:42:57.256779 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.256743    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:42:57.256890 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.256785    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a46d84-608a-428e-b5e0-c8337fb18da0-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:42:57.357914 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.357814    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:42:57.357914 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.357861    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:42:57.357914 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.357900    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:42:57.358222 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.357939    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sgf9\" (UniqueName: \"kubernetes.io/projected/48a46d84-608a-428e-b5e0-c8337fb18da0-kube-api-access-2sgf9\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:42:57.358222 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.358017    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:42:57.358222 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.358072    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a46d84-608a-428e-b5e0-c8337fb18da0-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:42:57.358612 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.358573    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:42:57.358728 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.358573    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:42:57.358728 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.358633    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:42:57.360328 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.360312    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:42:57.360692 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.360675    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a46d84-608a-428e-b5e0-c8337fb18da0-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:42:57.365780 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:42:57.365761    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sgf9\" (UniqueName: \"kubernetes.io/projected/48a46d84-608a-428e-b5e0-c8337fb18da0-kube-api-access-2sgf9\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:43:06.338904 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:43:06.338868    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:43:06.345201 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:43:06.345178    2566 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:43:06.440300 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:43:06.440263    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:43:06.440476 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:43:06.440416    2566 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:43:06.444577 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:43:06.444558    2566 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") DevicePath \"csi-0e8a7ed6a116a367a232c2eceb4606b3bc7c853843206c719654385d26f5ce26\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:43:06.535024 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:43:06.534985    2566 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/c95aa4f5833a65df4bc95280ad8355c13b05bf90eb1c331f50d3810708c6f2c6/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:43:06.552007 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:43:06.551973    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:43:06.710614 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:43:06.710580    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:43:06.851857 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:43:06.851828    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"]
Jul 27 17:43:06.853866 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:43:06.853833    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a46d84_608a_428e_b5e0_c8337fb18da0.slice/crio-6acbe232cf2454198e343d9cde0504233a5d9435135b5beba207a4059b9ae95d WatchSource:0}: Error finding container 6acbe232cf2454198e343d9cde0504233a5d9435135b5beba207a4059b9ae95d: Status 404 returned error can't find the container with id 6acbe232cf2454198e343d9cde0504233a5d9435135b5beba207a4059b9ae95d
Jul 27 17:43:07.511156 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:43:07.511110    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r" event={"ID":"48a46d84-608a-428e-b5e0-c8337fb18da0","Type":"ContainerStarted","Data":"5141e885a0a3b3530c7ca27c39bab17c3043cf8f95b6a6b6fcad6ebac4cf7b48"}
Jul 27 17:43:07.511643 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:43:07.511221    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r" event={"ID":"48a46d84-608a-428e-b5e0-c8337fb18da0","Type":"ContainerStarted","Data":"6acbe232cf2454198e343d9cde0504233a5d9435135b5beba207a4059b9ae95d"}
Jul 27 17:43:07.534482 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:43:07.534418    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r" podStartSLOduration=10.534386481 podStartE2EDuration="10.534386481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:43:07.530223644 +0000 UTC m=+2531.605488131" watchObservedRunningTime="2026-07-27 17:43:07.534386481 +0000 UTC m=+2531.609650971"
Jul 27 17:43:07.711282 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:43:07.711236    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:43:16.711325 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:43:16.711282    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:43:16.712924 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:43:16.712890    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r" podUID="48a46d84-608a-428e-b5e0-c8337fb18da0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 27 17:43:26.711906 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:43:26.711863    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r" podUID="48a46d84-608a-428e-b5e0-c8337fb18da0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 27 17:43:36.711838 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:43:36.711790    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r" podUID="48a46d84-608a-428e-b5e0-c8337fb18da0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 27 17:43:46.711374 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:43:46.711321    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r" podUID="48a46d84-608a-428e-b5e0-c8337fb18da0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 27 17:43:56.712027 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:43:56.711991    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r" podUID="48a46d84-608a-428e-b5e0-c8337fb18da0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 27 17:44:06.711532 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:44:06.711483    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r" podUID="48a46d84-608a-428e-b5e0-c8337fb18da0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 27 17:44:16.711992 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:44:16.711946    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r" podUID="48a46d84-608a-428e-b5e0-c8337fb18da0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 27 17:44:26.711707 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:44:26.711615    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r" podUID="48a46d84-608a-428e-b5e0-c8337fb18da0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 27 17:44:36.711615 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:44:36.711560    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r" podUID="48a46d84-608a-428e-b5e0-c8337fb18da0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 27 17:44:46.711482 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:44:46.711429    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r" podUID="48a46d84-608a-428e-b5e0-c8337fb18da0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 27 17:44:56.711736 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:44:56.711692    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r" podUID="48a46d84-608a-428e-b5e0-c8337fb18da0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 27 17:45:06.721222 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:06.721156    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:45:06.729043 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:06.729017    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:45:11.896282 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:11.896241    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"]
Jul 27 17:45:26.918796 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:26.918736    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r" podUID="48a46d84-608a-428e-b5e0-c8337fb18da0" containerName="main" containerID="cri-o://5141e885a0a3b3530c7ca27c39bab17c3043cf8f95b6a6b6fcad6ebac4cf7b48" gracePeriod=45
Jul 27 17:45:27.671089 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.671064    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:45:27.760044 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.759961    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-dshm\") pod \"48a46d84-608a-428e-b5e0-c8337fb18da0\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") "
Jul 27 17:45:27.760044 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.759997    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sgf9\" (UniqueName: \"kubernetes.io/projected/48a46d84-608a-428e-b5e0-c8337fb18da0-kube-api-access-2sgf9\") pod \"48a46d84-608a-428e-b5e0-c8337fb18da0\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") "
Jul 27 17:45:27.760311 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.760137    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") pod \"48a46d84-608a-428e-b5e0-c8337fb18da0\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") "
Jul 27 17:45:27.760311 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.760192    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-model-cache\") pod \"48a46d84-608a-428e-b5e0-c8337fb18da0\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") "
Jul 27 17:45:27.760311 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.760219    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-tmp-dir\") pod \"48a46d84-608a-428e-b5e0-c8337fb18da0\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") "
Jul 27 17:45:27.760311 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.760242    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a46d84-608a-428e-b5e0-c8337fb18da0-tls-certs\") pod \"48a46d84-608a-428e-b5e0-c8337fb18da0\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") "
Jul 27 17:45:27.760311 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.760282    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-home\") pod \"48a46d84-608a-428e-b5e0-c8337fb18da0\" (UID: \"48a46d84-608a-428e-b5e0-c8337fb18da0\") "
Jul 27 17:45:27.760573 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.760440    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-model-cache" (OuterVolumeSpecName: "model-cache") pod "48a46d84-608a-428e-b5e0-c8337fb18da0" (UID: "48a46d84-608a-428e-b5e0-c8337fb18da0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:45:27.760573 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.760565    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-model-cache\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:45:27.761056 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.761033    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-home" (OuterVolumeSpecName: "home") pod "48a46d84-608a-428e-b5e0-c8337fb18da0" (UID: "48a46d84-608a-428e-b5e0-c8337fb18da0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:45:27.762321 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.762298    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-dshm" (OuterVolumeSpecName: "dshm") pod "48a46d84-608a-428e-b5e0-c8337fb18da0" (UID: "48a46d84-608a-428e-b5e0-c8337fb18da0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:45:27.762598 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.762565    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a46d84-608a-428e-b5e0-c8337fb18da0-kube-api-access-2sgf9" (OuterVolumeSpecName: "kube-api-access-2sgf9") pod "48a46d84-608a-428e-b5e0-c8337fb18da0" (UID: "48a46d84-608a-428e-b5e0-c8337fb18da0"). InnerVolumeSpecName "kube-api-access-2sgf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:45:27.762779 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.762757    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a46d84-608a-428e-b5e0-c8337fb18da0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48a46d84-608a-428e-b5e0-c8337fb18da0" (UID: "48a46d84-608a-428e-b5e0-c8337fb18da0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:45:27.763214 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.763192    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f" (OuterVolumeSpecName: "kserve-pvc-source") pod "48a46d84-608a-428e-b5e0-c8337fb18da0" (UID: "48a46d84-608a-428e-b5e0-c8337fb18da0"). InnerVolumeSpecName "pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 27 17:45:27.771655 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.771631    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48a46d84-608a-428e-b5e0-c8337fb18da0" (UID: "48a46d84-608a-428e-b5e0-c8337fb18da0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:45:27.861534 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.861506    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-dshm\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:45:27.861534 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.861531    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sgf9\" (UniqueName: \"kubernetes.io/projected/48a46d84-608a-428e-b5e0-c8337fb18da0-kube-api-access-2sgf9\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:45:27.861684 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.861557    2566 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") on node \"ip-10-0-143-86.ec2.internal\" "
Jul 27 17:45:27.861684 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.861568    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-tmp-dir\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:45:27.861684 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.861577    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a46d84-608a-428e-b5e0-c8337fb18da0-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:45:27.861684 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.861584    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a46d84-608a-428e-b5e0-c8337fb18da0-home\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:45:27.890474 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.890455    2566 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f") on node "ip-10-0-143-86.ec2.internal"
Jul 27 17:45:27.962111 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:27.962076    2566 reconciler_common.go:299] "Volume detached for volume \"pvc-2d9890b1-c8bc-4f48-be9f-37fa271fca7c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f654f4e618cd940f\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"csi-0e8a7ed6a116a367a232c2eceb4606b3bc7c853843206c719654385d26f5ce26\""
Jul 27 17:45:28.315601 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:28.315569    2566 generic.go:356] "Generic (PLEG): container finished" podID="48a46d84-608a-428e-b5e0-c8337fb18da0" containerID="5141e885a0a3b3530c7ca27c39bab17c3043cf8f95b6a6b6fcad6ebac4cf7b48" exitCode=0
Jul 27 17:45:28.315798 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:28.315613    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"
Jul 27 17:45:28.315798 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:28.315617    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r" event={"ID":"48a46d84-608a-428e-b5e0-c8337fb18da0","Type":"ContainerDied","Data":"5141e885a0a3b3530c7ca27c39bab17c3043cf8f95b6a6b6fcad6ebac4cf7b48"}
Jul 27 17:45:28.315798 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:28.315736    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r" event={"ID":"48a46d84-608a-428e-b5e0-c8337fb18da0","Type":"ContainerDied","Data":"6acbe232cf2454198e343d9cde0504233a5d9435135b5beba207a4059b9ae95d"}
Jul 27 17:45:28.315798 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:28.315764    2566 scope.go:117] "RemoveContainer" containerID="5141e885a0a3b3530c7ca27c39bab17c3043cf8f95b6a6b6fcad6ebac4cf7b48"
Jul 27 17:45:28.332220 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:28.325822    2566 scope.go:117] "RemoveContainer" containerID="5141e885a0a3b3530c7ca27c39bab17c3043cf8f95b6a6b6fcad6ebac4cf7b48"
Jul 27 17:45:28.332665 ip-10-0-143-86 kubenswrapper[2566]: E0727 17:45:28.332516    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5141e885a0a3b3530c7ca27c39bab17c3043cf8f95b6a6b6fcad6ebac4cf7b48\": container with ID starting with 5141e885a0a3b3530c7ca27c39bab17c3043cf8f95b6a6b6fcad6ebac4cf7b48 not found: ID does not exist" containerID="5141e885a0a3b3530c7ca27c39bab17c3043cf8f95b6a6b6fcad6ebac4cf7b48"
Jul 27 17:45:28.332665 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:28.332569    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5141e885a0a3b3530c7ca27c39bab17c3043cf8f95b6a6b6fcad6ebac4cf7b48"} err="failed to get container status \"5141e885a0a3b3530c7ca27c39bab17c3043cf8f95b6a6b6fcad6ebac4cf7b48\": rpc error: code = NotFound desc = could not find container \"5141e885a0a3b3530c7ca27c39bab17c3043cf8f95b6a6b6fcad6ebac4cf7b48\": container with ID starting with 5141e885a0a3b3530c7ca27c39bab17c3043cf8f95b6a6b6fcad6ebac4cf7b48 not found: ID does not exist"
Jul 27 17:45:28.347874 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:28.347843    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"]
Jul 27 17:45:28.350024 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:28.349998    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcxd55r"]
Jul 27 17:45:28.403818 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:28.403786    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48a46d84-608a-428e-b5e0-c8337fb18da0" path="/var/lib/kubelet/pods/48a46d84-608a-428e-b5e0-c8337fb18da0/volumes"
Jul 27 17:45:35.095689 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:35.095654    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255"]
Jul 27 17:45:35.096156 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:35.096042    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48a46d84-608a-428e-b5e0-c8337fb18da0" containerName="main"
Jul 27 17:45:35.096156 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:35.096053    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="48a46d84-608a-428e-b5e0-c8337fb18da0" containerName="main"
Jul 27 17:45:35.096156 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:35.096118    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="48a46d84-608a-428e-b5e0-c8337fb18da0" containerName="main"
Jul 27 17:45:35.100848 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:35.100811    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255"
Jul 27 17:45:35.103684 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:35.103659    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 27 17:45:35.103684 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:35.103680    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-zh7c5\""
Jul 27 17:45:35.103907 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:35.103726    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255"]
Jul 27 17:45:35.104513 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:35.104492    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 27 17:45:35.125300 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:35.125273    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwv97\" (UniqueName: \"kubernetes.io/projected/561cf276-c5a0-41f1-abb3-8837aa7a9aff-kube-api-access-pwv97\") pod \"e2e-pvc-model-download-m4255\" (UID: \"561cf276-c5a0-41f1-abb3-8837aa7a9aff\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255"
Jul 27 17:45:35.225938 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:35.225894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwv97\" (UniqueName: \"kubernetes.io/projected/561cf276-c5a0-41f1-abb3-8837aa7a9aff-kube-api-access-pwv97\") pod \"e2e-pvc-model-download-m4255\" (UID: \"561cf276-c5a0-41f1-abb3-8837aa7a9aff\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255"
Jul 27 17:45:35.234793 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:35.234754    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwv97\" (UniqueName: \"kubernetes.io/projected/561cf276-c5a0-41f1-abb3-8837aa7a9aff-kube-api-access-pwv97\") pod \"e2e-pvc-model-download-m4255\" (UID: \"561cf276-c5a0-41f1-abb3-8837aa7a9aff\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255"
Jul 27 17:45:36.830273 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:36.830237    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq"]
Jul 27 17:45:36.835186 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:36.835149    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-hgsqq"]
Jul 27 17:45:38.403437 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:38.403402    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea1882ee-2f37-4838-ac7a-004f42e93893" path="/var/lib/kubelet/pods/ea1882ee-2f37-4838-ac7a-004f42e93893/volumes"
Jul 27 17:45:39.261426 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:39.261391    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-56638b18-3f61-49c9-8c07-2721db06574b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03f4d5a28743889ed\") pod \"e2e-pvc-model-download-m4255\" (UID: \"561cf276-c5a0-41f1-abb3-8837aa7a9aff\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255"
Jul 27 17:45:39.267096 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:39.267073    2566 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-56638b18-3f61-49c9-8c07-2721db06574b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03f4d5a28743889ed\") pod \"e2e-pvc-model-download-m4255\" (UID: \"561cf276-c5a0-41f1-abb3-8837aa7a9aff\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255"
Jul 27 17:45:39.362620 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:39.362582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-56638b18-3f61-49c9-8c07-2721db06574b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03f4d5a28743889ed\") pod \"e2e-pvc-model-download-m4255\" (UID: \"561cf276-c5a0-41f1-abb3-8837aa7a9aff\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255"
Jul 27 17:45:39.362794 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:39.362708    2566 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-56638b18-3f61-49c9-8c07-2721db06574b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03f4d5a28743889ed\") pod \"e2e-pvc-model-download-m4255\" (UID: \"561cf276-c5a0-41f1-abb3-8837aa7a9aff\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255"
Jul 27 17:45:39.365860 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:39.365838    2566 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-56638b18-3f61-49c9-8c07-2721db06574b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03f4d5a28743889ed\") pod \"e2e-pvc-model-download-m4255\" (UID: \"561cf276-c5a0-41f1-abb3-8837aa7a9aff\") DevicePath \"csi-cf909ba9ccf0434be837965c3d491f51f0b6187d703013bb433c17cc3db357d5\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255"
Jul 27 17:45:39.577927 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:39.577894    2566 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-56638b18-3f61-49c9-8c07-2721db06574b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03f4d5a28743889ed\") pod \"e2e-pvc-model-download-m4255\" (UID: \"561cf276-c5a0-41f1-abb3-8837aa7a9aff\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b2c97a5f1a1a5d12e220d6a17e7f84e8ca993c6a9488cdbe36f4e48b5fc72ce2/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255"
Jul 27 17:45:39.590917 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:39.590892    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-56638b18-3f61-49c9-8c07-2721db06574b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03f4d5a28743889ed\") pod \"e2e-pvc-model-download-m4255\" (UID: \"561cf276-c5a0-41f1-abb3-8837aa7a9aff\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255"
Jul 27 17:45:39.611817 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:39.611787    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255"
Jul 27 17:45:39.743407 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:39.743378    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255"]
Jul 27 17:45:39.745301 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:45:39.745276    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod561cf276_c5a0_41f1_abb3_8837aa7a9aff.slice/crio-bd6a31ccd32722c885a3b03962759bba9c363a4853f1ccdb0e242bb9ad8462e9 WatchSource:0}: Error finding container bd6a31ccd32722c885a3b03962759bba9c363a4853f1ccdb0e242bb9ad8462e9: Status 404 returned error can't find the container with id bd6a31ccd32722c885a3b03962759bba9c363a4853f1ccdb0e242bb9ad8462e9
Jul 27 17:45:40.388253 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:40.388213    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255" event={"ID":"561cf276-c5a0-41f1-abb3-8837aa7a9aff","Type":"ContainerStarted","Data":"6e053d7cf31f51405c12048cedc6b9c065c5cd7925111d447dbc4d530cc1f2bd"}
Jul 27 17:45:40.388455 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:40.388264    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255" event={"ID":"561cf276-c5a0-41f1-abb3-8837aa7a9aff","Type":"ContainerStarted","Data":"bd6a31ccd32722c885a3b03962759bba9c363a4853f1ccdb0e242bb9ad8462e9"}
Jul 27 17:45:40.403189 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:40.403124    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255" podStartSLOduration=8.403110605 podStartE2EDuration="8.403110605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:45:40.402024293 +0000 UTC m=+2684.477288780" watchObservedRunningTime="2026-07-27 17:45:40.403110605 +0000 UTC m=+2684.478375092"
Jul 27 17:45:49.440811 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:49.440774    2566 generic.go:356] "Generic (PLEG): container finished" podID="561cf276-c5a0-41f1-abb3-8837aa7a9aff" containerID="6e053d7cf31f51405c12048cedc6b9c065c5cd7925111d447dbc4d530cc1f2bd" exitCode=0
Jul 27 17:45:49.441258 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:49.440848    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255" event={"ID":"561cf276-c5a0-41f1-abb3-8837aa7a9aff","Type":"ContainerDied","Data":"6e053d7cf31f51405c12048cedc6b9c065c5cd7925111d447dbc4d530cc1f2bd"}
Jul 27 17:45:50.594461 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:50.594433    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255"
Jul 27 17:45:50.669559 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:50.669528    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03f4d5a28743889ed\") pod \"561cf276-c5a0-41f1-abb3-8837aa7a9aff\" (UID: \"561cf276-c5a0-41f1-abb3-8837aa7a9aff\") "
Jul 27 17:45:50.669715 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:50.669567    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwv97\" (UniqueName: \"kubernetes.io/projected/561cf276-c5a0-41f1-abb3-8837aa7a9aff-kube-api-access-pwv97\") pod \"561cf276-c5a0-41f1-abb3-8837aa7a9aff\" (UID: \"561cf276-c5a0-41f1-abb3-8837aa7a9aff\") "
Jul 27 17:45:50.671796 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:50.671770    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/561cf276-c5a0-41f1-abb3-8837aa7a9aff-kube-api-access-pwv97" (OuterVolumeSpecName: "kube-api-access-pwv97") pod "561cf276-c5a0-41f1-abb3-8837aa7a9aff" (UID: "561cf276-c5a0-41f1-abb3-8837aa7a9aff"). InnerVolumeSpecName "kube-api-access-pwv97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:45:50.672274 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:50.672248    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03f4d5a28743889ed" (OuterVolumeSpecName: "model-storage") pod "561cf276-c5a0-41f1-abb3-8837aa7a9aff" (UID: "561cf276-c5a0-41f1-abb3-8837aa7a9aff"). InnerVolumeSpecName "pvc-56638b18-3f61-49c9-8c07-2721db06574b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 27 17:45:50.770976 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:50.770903    2566 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-56638b18-3f61-49c9-8c07-2721db06574b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03f4d5a28743889ed\") on node \"ip-10-0-143-86.ec2.internal\" "
Jul 27 17:45:50.770976 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:50.770931    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwv97\" (UniqueName: \"kubernetes.io/projected/561cf276-c5a0-41f1-abb3-8837aa7a9aff-kube-api-access-pwv97\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 17:45:50.892703 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:50.892680    2566 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-56638b18-3f61-49c9-8c07-2721db06574b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-03f4d5a28743889ed") on node "ip-10-0-143-86.ec2.internal"
Jul 27 17:45:50.972405 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:50.972372    2566 reconciler_common.go:299] "Volume detached for volume \"pvc-56638b18-3f61-49c9-8c07-2721db06574b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03f4d5a28743889ed\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"csi-cf909ba9ccf0434be837965c3d491f51f0b6187d703013bb433c17cc3db357d5\""
Jul 27 17:45:51.452151 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:51.452119    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255"
Jul 27 17:45:51.452151 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:51.452137    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4255" event={"ID":"561cf276-c5a0-41f1-abb3-8837aa7a9aff","Type":"ContainerDied","Data":"bd6a31ccd32722c885a3b03962759bba9c363a4853f1ccdb0e242bb9ad8462e9"}
Jul 27 17:45:51.452449 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:51.452189    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd6a31ccd32722c885a3b03962759bba9c363a4853f1ccdb0e242bb9ad8462e9"
Jul 27 17:45:57.590002 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:57.589971    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:45:57.599020 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:45:57.598998    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:46:16.027836 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.027800    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"]
Jul 27 17:46:16.028315 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.028237    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="561cf276-c5a0-41f1-abb3-8837aa7a9aff" containerName="storage-initializer"
Jul 27 17:46:16.028315 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.028252    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="561cf276-c5a0-41f1-abb3-8837aa7a9aff" containerName="storage-initializer"
Jul 27 17:46:16.028393 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.028317    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="561cf276-c5a0-41f1-abb3-8837aa7a9aff" containerName="storage-initializer"
Jul 27 17:46:16.031672 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.031650    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:16.034386 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.034360    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 27 17:46:16.034539 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.034505    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 27 17:46:16.035575 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.035549    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-zh7c5\""
Jul 27 17:46:16.036249 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.036161    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-l4jvn\""
Jul 27 17:46:16.036558 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.036540    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 27 17:46:16.044326 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.044295    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"]
Jul 27 17:46:16.192368 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.192333    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/232e8c6b-3861-4cf4-b14c-cd5777e84c58-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:16.192546 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.192380    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/232e8c6b-3861-4cf4-b14c-cd5777e84c58-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:16.192546 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.192408    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/232e8c6b-3861-4cf4-b14c-cd5777e84c58-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:16.192546 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.192450    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/232e8c6b-3861-4cf4-b14c-cd5777e84c58-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:16.192546 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.192514    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/232e8c6b-3861-4cf4-b14c-cd5777e84c58-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:16.192546 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.192532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dlts\" (UniqueName: \"kubernetes.io/projected/232e8c6b-3861-4cf4-b14c-cd5777e84c58-kube-api-access-5dlts\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:16.293552 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.293457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/232e8c6b-3861-4cf4-b14c-cd5777e84c58-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:16.293552 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.293506    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/232e8c6b-3861-4cf4-b14c-cd5777e84c58-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:16.293552 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.293535    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/232e8c6b-3861-4cf4-b14c-cd5777e84c58-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:16.293552 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.293556    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/232e8c6b-3861-4cf4-b14c-cd5777e84c58-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:16.293886 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.293604    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/232e8c6b-3861-4cf4-b14c-cd5777e84c58-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:16.293886 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.293623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dlts\" (UniqueName: \"kubernetes.io/projected/232e8c6b-3861-4cf4-b14c-cd5777e84c58-kube-api-access-5dlts\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:16.294020 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.293945    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/232e8c6b-3861-4cf4-b14c-cd5777e84c58-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:16.294020 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.293967    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/232e8c6b-3861-4cf4-b14c-cd5777e84c58-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:16.294020 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.294003    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/232e8c6b-3861-4cf4-b14c-cd5777e84c58-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:16.295989 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.295967    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/232e8c6b-3861-4cf4-b14c-cd5777e84c58-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:16.296150 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.296134    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/232e8c6b-3861-4cf4-b14c-cd5777e84c58-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:16.301791 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:16.301765    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dlts\" (UniqueName: \"kubernetes.io/projected/232e8c6b-3861-4cf4-b14c-cd5777e84c58-kube-api-access-5dlts\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:19.929363 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:19.929329    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-56638b18-3f61-49c9-8c07-2721db06574b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03f4d5a28743889ed\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:19.934998 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:19.934973    2566 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-56638b18-3f61-49c9-8c07-2721db06574b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03f4d5a28743889ed\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:20.030778 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:20.030743    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-56638b18-3f61-49c9-8c07-2721db06574b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03f4d5a28743889ed\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:20.030936 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:20.030888    2566 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-56638b18-3f61-49c9-8c07-2721db06574b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03f4d5a28743889ed\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:20.033555 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:20.033531    2566 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-56638b18-3f61-49c9-8c07-2721db06574b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03f4d5a28743889ed\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") DevicePath \"csi-cf909ba9ccf0434be837965c3d491f51f0b6187d703013bb433c17cc3db357d5\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:20.118618 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:20.118583    2566 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-56638b18-3f61-49c9-8c07-2721db06574b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03f4d5a28743889ed\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b2c97a5f1a1a5d12e220d6a17e7f84e8ca993c6a9488cdbe36f4e48b5fc72ce2/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:20.135464 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:20.135434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-56638b18-3f61-49c9-8c07-2721db06574b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03f4d5a28743889ed\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582\" (UID: \"232e8c6b-3861-4cf4-b14c-cd5777e84c58\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:20.244155 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:20.244120    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:20.387568 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:20.387537    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"]
Jul 27 17:46:20.389307 ip-10-0-143-86 kubenswrapper[2566]: W0727 17:46:20.389283    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232e8c6b_3861_4cf4_b14c_cd5777e84c58.slice/crio-370be5cff3ba22a9aa14264cc8c7e994d7a9ffeb027a7d342c982e8c92e359a6 WatchSource:0}: Error finding container 370be5cff3ba22a9aa14264cc8c7e994d7a9ffeb027a7d342c982e8c92e359a6: Status 404 returned error can't find the container with id 370be5cff3ba22a9aa14264cc8c7e994d7a9ffeb027a7d342c982e8c92e359a6
Jul 27 17:46:20.391571 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:20.391555    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:46:20.615028 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:20.614939    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582" event={"ID":"232e8c6b-3861-4cf4-b14c-cd5777e84c58","Type":"ContainerStarted","Data":"370be5cff3ba22a9aa14264cc8c7e994d7a9ffeb027a7d342c982e8c92e359a6"}
Jul 27 17:46:23.634871 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:23.634824    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582" event={"ID":"232e8c6b-3861-4cf4-b14c-cd5777e84c58","Type":"ContainerStarted","Data":"49cfeb0e7e8481948c2362703626b137f72797d55b3dfe4bd1891154d2c791b7"}
Jul 27 17:46:23.635307 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:23.634903    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:24.643140 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:24.643096    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582" event={"ID":"232e8c6b-3861-4cf4-b14c-cd5777e84c58","Type":"ContainerStarted","Data":"ee76192acf3634911c7b5c4af7465ca46d15f421cac06afaabaf83df23421594"}
Jul 27 17:46:24.667077 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:24.667013    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582" podStartSLOduration=6.247473632 podStartE2EDuration="8.666994296s" totalImagesPullingTime="2.419520664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:46:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:46:24.663758407 +0000 UTC m=+2728.739022905" watchObservedRunningTime="2026-07-27 17:46:24.666994296 +0000 UTC m=+2728.742258783"
Jul 27 17:46:30.244349 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:30.244300    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:30.244888 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:30.244366    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:30.245779 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:30.245748    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582" podUID="232e8c6b-3861-4cf4-b14c-cd5777e84c58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8001/health\": dial tcp 10.132.0.64:8001: connect: connection refused"
Jul 27 17:46:40.245346 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:40.245296    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582" podUID="232e8c6b-3861-4cf4-b14c-cd5777e84c58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8001/health\": dial tcp 10.132.0.64:8001: connect: connection refused"
Jul 27 17:46:40.257101 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:40.257074    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:46:50.245530 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:46:50.245472    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582" podUID="232e8c6b-3861-4cf4-b14c-cd5777e84c58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8001/health\": dial tcp 10.132.0.64:8001: connect: connection refused"
Jul 27 17:47:00.244703 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:47:00.244646    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582" podUID="232e8c6b-3861-4cf4-b14c-cd5777e84c58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8001/health\": dial tcp 10.132.0.64:8001: connect: connection refused"
Jul 27 17:47:10.245346 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:47:10.245286    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582" podUID="232e8c6b-3861-4cf4-b14c-cd5777e84c58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8001/health\": dial tcp 10.132.0.64:8001: connect: connection refused"
Jul 27 17:47:20.245085 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:47:20.244976    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582" podUID="232e8c6b-3861-4cf4-b14c-cd5777e84c58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8001/health\": dial tcp 10.132.0.64:8001: connect: connection refused"
Jul 27 17:47:30.245203 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:47:30.245128    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582" podUID="232e8c6b-3861-4cf4-b14c-cd5777e84c58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8001/health\": dial tcp 10.132.0.64:8001: connect: connection refused"
Jul 27 17:47:40.245317 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:47:40.245268    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582" podUID="232e8c6b-3861-4cf4-b14c-cd5777e84c58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8001/health\": dial tcp 10.132.0.64:8001: connect: connection refused"
Jul 27 17:47:50.245387 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:47:50.245328    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582" podUID="232e8c6b-3861-4cf4-b14c-cd5777e84c58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8001/health\": dial tcp 10.132.0.64:8001: connect: connection refused"
Jul 27 17:48:00.245001 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:48:00.244931    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582" podUID="232e8c6b-3861-4cf4-b14c-cd5777e84c58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8001/health\": dial tcp 10.132.0.64:8001: connect: connection refused"
Jul 27 17:48:10.245501 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:48:10.245452    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582" podUID="232e8c6b-3861-4cf4-b14c-cd5777e84c58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8001/health\": dial tcp 10.132.0.64:8001: connect: connection refused"
Jul 27 17:48:20.254860 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:48:20.254826    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:48:20.266246 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:48:20.266217    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582"
Jul 27 17:48:56.859818 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:48:56.859780    2566 scope.go:117] "RemoveContainer" containerID="d06ccfa7c06e0186f7639bd29476456bf1da895e32c331f3f5470981601705ad"
Jul 27 17:50:57.629695 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:50:57.629663    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:50:57.639793 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:50:57.639769    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:55:57.672504 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:55:57.672393    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 17:55:57.682811 ip-10-0-143-86 kubenswrapper[2566]: I0727 17:55:57.682787    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 18:00:53.285141 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:00:53.285100    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582_232e8c6b-3861-4cf4-b14c-cd5777e84c58/llm-d-routing-sidecar/0.log"
Jul 27 18:00:53.296447 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:00:53.296417    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499ffr582_232e8c6b-3861-4cf4-b14c-cd5777e84c58/main/0.log"
Jul 27 18:00:57.711712 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:00:57.711681    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 18:00:57.727870 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:00:57.727848    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 18:00:58.488527 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:00:58.488494    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599"]
Jul 27 18:00:58.491905 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:00:58.491887    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599"
Jul 27 18:00:58.494219 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:00:58.494201    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 27 18:00:58.495140 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:00:58.495115    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 27 18:00:58.495262 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:00:58.495206    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-4r6xp\""
Jul 27 18:00:58.507819 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:00:58.507800    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599"]
Jul 27 18:00:58.544192 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:00:58.544142    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4krgj\" (UniqueName: \"kubernetes.io/projected/0b5d612c-35d6-417e-82d9-3b9aacaae5c6-kube-api-access-4krgj\") pod \"e2e-pvc-model-download-ts599\" (UID: \"0b5d612c-35d6-417e-82d9-3b9aacaae5c6\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599"
Jul 27 18:00:58.645728 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:00:58.645689    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4krgj\" (UniqueName: \"kubernetes.io/projected/0b5d612c-35d6-417e-82d9-3b9aacaae5c6-kube-api-access-4krgj\") pod \"e2e-pvc-model-download-ts599\" (UID: \"0b5d612c-35d6-417e-82d9-3b9aacaae5c6\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599"
Jul 27 18:00:58.654120 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:00:58.654089    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4krgj\" (UniqueName: \"kubernetes.io/projected/0b5d612c-35d6-417e-82d9-3b9aacaae5c6-kube-api-access-4krgj\") pod \"e2e-pvc-model-download-ts599\" (UID: \"0b5d612c-35d6-417e-82d9-3b9aacaae5c6\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599"
Jul 27 18:01:08.939833 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:01:08.939735    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") pod \"e2e-pvc-model-download-ts599\" (UID: \"0b5d612c-35d6-417e-82d9-3b9aacaae5c6\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599"
Jul 27 18:01:08.945427 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:01:08.945399    2566 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") pod \"e2e-pvc-model-download-ts599\" (UID: \"0b5d612c-35d6-417e-82d9-3b9aacaae5c6\") device path: \"\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599"
Jul 27 18:01:09.040821 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:01:09.040790    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") pod \"e2e-pvc-model-download-ts599\" (UID: \"0b5d612c-35d6-417e-82d9-3b9aacaae5c6\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599"
Jul 27 18:01:09.040986 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:01:09.040926    2566 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") pod \"e2e-pvc-model-download-ts599\" (UID: \"0b5d612c-35d6-417e-82d9-3b9aacaae5c6\") DevicePath \"\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599"
Jul 27 18:01:09.044153 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:01:09.044128    2566 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") pod \"e2e-pvc-model-download-ts599\" (UID: \"0b5d612c-35d6-417e-82d9-3b9aacaae5c6\") DevicePath \"csi-630dddfe5bbcc0ec2bfe0a63a4fad07796b940bce0091ad7473a8830e836a5ae\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599"
Jul 27 18:01:09.284751 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:01:09.284719    2566 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") pod \"e2e-pvc-model-download-ts599\" (UID: \"0b5d612c-35d6-417e-82d9-3b9aacaae5c6\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d6958b6c657add15825891a2101a701b3f00445af44cee5d41c15920ade36472/globalmount\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599"
Jul 27 18:01:09.298241 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:01:09.298215    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") pod \"e2e-pvc-model-download-ts599\" (UID: \"0b5d612c-35d6-417e-82d9-3b9aacaae5c6\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599"
Jul 27 18:01:09.600740 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:01:09.600652    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599"
Jul 27 18:01:09.734203 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:01:09.734156    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599"]
Jul 27 18:01:09.735904 ip-10-0-143-86 kubenswrapper[2566]: W0727 18:01:09.735876    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b5d612c_35d6_417e_82d9_3b9aacaae5c6.slice/crio-1ab259ea339f564180e99f75572172c2022a542ca1ad2e3b07d4a259b6dc911b WatchSource:0}: Error finding container 1ab259ea339f564180e99f75572172c2022a542ca1ad2e3b07d4a259b6dc911b: Status 404 returned error can't find the container with id 1ab259ea339f564180e99f75572172c2022a542ca1ad2e3b07d4a259b6dc911b
Jul 27 18:01:10.515789 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:01:10.515752    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599" event={"ID":"0b5d612c-35d6-417e-82d9-3b9aacaae5c6","Type":"ContainerStarted","Data":"90166070deccbe004653d823a62f4ba5df43db8a63bf88525f7f209b98055131"}
Jul 27 18:01:10.516261 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:01:10.515797    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599" event={"ID":"0b5d612c-35d6-417e-82d9-3b9aacaae5c6","Type":"ContainerStarted","Data":"1ab259ea339f564180e99f75572172c2022a542ca1ad2e3b07d4a259b6dc911b"}
Jul 27 18:01:10.536188 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:01:10.534969    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599" podStartSLOduration=16.534945963 podStartE2EDuration="16.534945963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:01:10.530436864 +0000 UTC m=+3614.605701353" watchObservedRunningTime="2026-07-27 18:01:10.534945963 +0000 UTC m=+3614.610210452"
Jul 27 18:02:49.060330 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:02:49.060237    2566 generic.go:356] "Generic (PLEG): container finished" podID="0b5d612c-35d6-417e-82d9-3b9aacaae5c6" containerID="90166070deccbe004653d823a62f4ba5df43db8a63bf88525f7f209b98055131" exitCode=0
Jul 27 18:02:49.060794 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:02:49.060314    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599" event={"ID":"0b5d612c-35d6-417e-82d9-3b9aacaae5c6","Type":"ContainerDied","Data":"90166070deccbe004653d823a62f4ba5df43db8a63bf88525f7f209b98055131"}
Jul 27 18:02:50.214787 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:02:50.214753    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599"
Jul 27 18:02:50.247648 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:02:50.247618    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4krgj\" (UniqueName: \"kubernetes.io/projected/0b5d612c-35d6-417e-82d9-3b9aacaae5c6-kube-api-access-4krgj\") pod \"0b5d612c-35d6-417e-82d9-3b9aacaae5c6\" (UID: \"0b5d612c-35d6-417e-82d9-3b9aacaae5c6\") "
Jul 27 18:02:50.247828 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:02:50.247814    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") pod \"0b5d612c-35d6-417e-82d9-3b9aacaae5c6\" (UID: \"0b5d612c-35d6-417e-82d9-3b9aacaae5c6\") "
Jul 27 18:02:50.250131 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:02:50.250102    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5d612c-35d6-417e-82d9-3b9aacaae5c6-kube-api-access-4krgj" (OuterVolumeSpecName: "kube-api-access-4krgj") pod "0b5d612c-35d6-417e-82d9-3b9aacaae5c6" (UID: "0b5d612c-35d6-417e-82d9-3b9aacaae5c6"). InnerVolumeSpecName "kube-api-access-4krgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:02:50.250741 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:02:50.250719    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb" (OuterVolumeSpecName: "model-storage") pod "0b5d612c-35d6-417e-82d9-3b9aacaae5c6" (UID: "0b5d612c-35d6-417e-82d9-3b9aacaae5c6"). InnerVolumeSpecName "pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 27 18:02:50.348620 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:02:50.348525    2566 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") on node \"ip-10-0-143-86.ec2.internal\" "
Jul 27 18:02:50.348620 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:02:50.348562    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4krgj\" (UniqueName: \"kubernetes.io/projected/0b5d612c-35d6-417e-82d9-3b9aacaae5c6-kube-api-access-4krgj\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 18:02:50.456434 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:02:50.456410    2566 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb") on node "ip-10-0-143-86.ec2.internal"
Jul 27 18:02:50.551241 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:02:50.551202    2566 reconciler_common.go:299] "Volume detached for volume \"pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"csi-630dddfe5bbcc0ec2bfe0a63a4fad07796b940bce0091ad7473a8830e836a5ae\""
Jul 27 18:02:51.072373 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:02:51.072343    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599"
Jul 27 18:02:51.072548 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:02:51.072376    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599" event={"ID":"0b5d612c-35d6-417e-82d9-3b9aacaae5c6","Type":"ContainerDied","Data":"1ab259ea339f564180e99f75572172c2022a542ca1ad2e3b07d4a259b6dc911b"}
Jul 27 18:02:51.072548 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:02:51.072415    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ab259ea339f564180e99f75572172c2022a542ca1ad2e3b07d4a259b6dc911b"
Jul 27 18:03:14.255588 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.255551    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 27 18:03:14.256086 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.255938    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5d612c-35d6-417e-82d9-3b9aacaae5c6" containerName="storage-initializer"
Jul 27 18:03:14.256086 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.255950    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5d612c-35d6-417e-82d9-3b9aacaae5c6" containerName="storage-initializer"
Jul 27 18:03:14.256086 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.256022    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b5d612c-35d6-417e-82d9-3b9aacaae5c6" containerName="storage-initializer"
Jul 27 18:03:14.259283 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.259264    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:14.264945 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.264918    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 27 18:03:14.265051 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.265001    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-85z6x\""
Jul 27 18:03:14.265822 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.265802    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 27 18:03:14.265920 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.265802    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-4r6xp\""
Jul 27 18:03:14.266327 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.266104    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 27 18:03:14.272821 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.272785    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 27 18:03:14.362721 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.362688    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:14.362903 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.362727    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:14.362903 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.362754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:14.362903 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.362836    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5deb78a3-7dca-485c-801c-7a4188e42108-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:14.363058 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.362897    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqrs6\" (UniqueName: \"kubernetes.io/projected/5deb78a3-7dca-485c-801c-7a4188e42108-kube-api-access-nqrs6\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:14.363058 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.362949    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:14.464007 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.463975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:14.464007 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.464014    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:14.464264 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.464142    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:14.464264 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.464235    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5deb78a3-7dca-485c-801c-7a4188e42108-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:14.464379 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.464277    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqrs6\" (UniqueName: \"kubernetes.io/projected/5deb78a3-7dca-485c-801c-7a4188e42108-kube-api-access-nqrs6\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:14.464379 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.464335    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:14.464379 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.464356    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:14.464556 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.464462    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:14.464656 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.464640    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:14.466314 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.466288    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:14.466754 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.466739    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5deb78a3-7dca-485c-801c-7a4188e42108-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:14.477667 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:14.477635    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqrs6\" (UniqueName: \"kubernetes.io/projected/5deb78a3-7dca-485c-801c-7a4188e42108-kube-api-access-nqrs6\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:21.228728 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:21.228690    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:21.234798 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:21.234773    2566 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") device path: \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:21.330154 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:21.330060    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:21.330357 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:21.330205    2566 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") DevicePath \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:21.333441 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:21.333421    2566 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") DevicePath \"csi-630dddfe5bbcc0ec2bfe0a63a4fad07796b940bce0091ad7473a8830e836a5ae\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:21.423647 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:21.423615    2566 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d6958b6c657add15825891a2101a701b3f00445af44cee5d41c15920ade36472/globalmount\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:21.435306 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:21.435280    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:21.470509 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:21.470476    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:21.610317 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:21.610271    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 27 18:03:21.612762 ip-10-0-143-86 kubenswrapper[2566]: W0727 18:03:21.612734    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5deb78a3_7dca_485c_801c_7a4188e42108.slice/crio-bab03da3436a2577c429ac065a03d56f58ed321cf39ae170bfbc06ca8b8fb186 WatchSource:0}: Error finding container bab03da3436a2577c429ac065a03d56f58ed321cf39ae170bfbc06ca8b8fb186: Status 404 returned error can't find the container with id bab03da3436a2577c429ac065a03d56f58ed321cf39ae170bfbc06ca8b8fb186
Jul 27 18:03:22.244610 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:22.244575    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"5deb78a3-7dca-485c-801c-7a4188e42108","Type":"ContainerStarted","Data":"ff9ba07fa54244d135e271ef129cdec491f5831e3058797e243ecb05bc3be71a"}
Jul 27 18:03:22.245086 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:22.244643    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"5deb78a3-7dca-485c-801c-7a4188e42108","Type":"ContainerStarted","Data":"bab03da3436a2577c429ac065a03d56f58ed321cf39ae170bfbc06ca8b8fb186"}
Jul 27 18:03:22.268729 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:22.268664    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podStartSLOduration=8.268645363 podStartE2EDuration="8.268645363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:03:22.264227758 +0000 UTC m=+3746.339492239" watchObservedRunningTime="2026-07-27 18:03:22.268645363 +0000 UTC m=+3746.343909854"
Jul 27 18:03:22.470888 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:22.470852    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:31.471232 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:31.471188    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:03:31.472953 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:31.472923    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="5deb78a3-7dca-485c-801c-7a4188e42108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 27 18:03:41.471262 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:41.471220    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="5deb78a3-7dca-485c-801c-7a4188e42108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 27 18:03:51.471026 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:03:51.470977    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="5deb78a3-7dca-485c-801c-7a4188e42108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 27 18:04:01.471232 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:04:01.471179    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="5deb78a3-7dca-485c-801c-7a4188e42108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 27 18:04:11.471573 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:04:11.471524    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="5deb78a3-7dca-485c-801c-7a4188e42108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 27 18:04:21.471154 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:04:21.471105    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="5deb78a3-7dca-485c-801c-7a4188e42108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 27 18:04:31.471345 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:04:31.471293    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="5deb78a3-7dca-485c-801c-7a4188e42108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 27 18:04:41.471147 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:04:41.471099    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="5deb78a3-7dca-485c-801c-7a4188e42108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 27 18:04:51.471478 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:04:51.471428    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="5deb78a3-7dca-485c-801c-7a4188e42108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 27 18:05:01.481065 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:01.481026    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:05:01.488605 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:01.488580    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:05:38.939195 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:38.939088    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 27 18:05:53.961742 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:53.961663    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="5deb78a3-7dca-485c-801c-7a4188e42108" containerName="main" containerID="cri-o://ff9ba07fa54244d135e271ef129cdec491f5831e3058797e243ecb05bc3be71a" gracePeriod=45
Jul 27 18:05:54.482184 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.482128    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="5deb78a3-7dca-485c-801c-7a4188e42108" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 27 18:05:54.730952 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.730924    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:05:54.810207 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.810104    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") pod \"5deb78a3-7dca-485c-801c-7a4188e42108\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") "
Jul 27 18:05:54.810207 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.810181    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5deb78a3-7dca-485c-801c-7a4188e42108-tls-certs\") pod \"5deb78a3-7dca-485c-801c-7a4188e42108\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") "
Jul 27 18:05:54.810207 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.810209    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-home\") pod \"5deb78a3-7dca-485c-801c-7a4188e42108\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") "
Jul 27 18:05:54.810482 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.810278    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-dshm\") pod \"5deb78a3-7dca-485c-801c-7a4188e42108\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") "
Jul 27 18:05:54.810482 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.810299    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-tmp-dir\") pod \"5deb78a3-7dca-485c-801c-7a4188e42108\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") "
Jul 27 18:05:54.810482 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.810323    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-model-cache\") pod \"5deb78a3-7dca-485c-801c-7a4188e42108\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") "
Jul 27 18:05:54.810482 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.810350    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqrs6\" (UniqueName: \"kubernetes.io/projected/5deb78a3-7dca-485c-801c-7a4188e42108-kube-api-access-nqrs6\") pod \"5deb78a3-7dca-485c-801c-7a4188e42108\" (UID: \"5deb78a3-7dca-485c-801c-7a4188e42108\") "
Jul 27 18:05:54.810751 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.810723    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-model-cache" (OuterVolumeSpecName: "model-cache") pod "5deb78a3-7dca-485c-801c-7a4188e42108" (UID: "5deb78a3-7dca-485c-801c-7a4188e42108"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:05:54.811054 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.811010    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-home" (OuterVolumeSpecName: "home") pod "5deb78a3-7dca-485c-801c-7a4188e42108" (UID: "5deb78a3-7dca-485c-801c-7a4188e42108"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:05:54.812876 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.812839    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-dshm" (OuterVolumeSpecName: "dshm") pod "5deb78a3-7dca-485c-801c-7a4188e42108" (UID: "5deb78a3-7dca-485c-801c-7a4188e42108"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:05:54.812995 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.812931    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5deb78a3-7dca-485c-801c-7a4188e42108-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5deb78a3-7dca-485c-801c-7a4188e42108" (UID: "5deb78a3-7dca-485c-801c-7a4188e42108"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:05:54.813188 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.813139    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5deb78a3-7dca-485c-801c-7a4188e42108-kube-api-access-nqrs6" (OuterVolumeSpecName: "kube-api-access-nqrs6") pod "5deb78a3-7dca-485c-801c-7a4188e42108" (UID: "5deb78a3-7dca-485c-801c-7a4188e42108"). InnerVolumeSpecName "kube-api-access-nqrs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:05:54.813562 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.813542    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb" (OuterVolumeSpecName: "kserve-pvc-source") pod "5deb78a3-7dca-485c-801c-7a4188e42108" (UID: "5deb78a3-7dca-485c-801c-7a4188e42108"). InnerVolumeSpecName "pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 27 18:05:54.823367 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.823341    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5deb78a3-7dca-485c-801c-7a4188e42108" (UID: "5deb78a3-7dca-485c-801c-7a4188e42108"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:05:54.912005 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.911970    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-dshm\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 18:05:54.912005 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.912000    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-tmp-dir\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 18:05:54.912005 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.912010    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-model-cache\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 18:05:54.912284 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.912020    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqrs6\" (UniqueName: \"kubernetes.io/projected/5deb78a3-7dca-485c-801c-7a4188e42108-kube-api-access-nqrs6\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 18:05:54.912284 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.912046    2566 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") on node \"ip-10-0-143-86.ec2.internal\" "
Jul 27 18:05:54.912284 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.912056    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5deb78a3-7dca-485c-801c-7a4188e42108-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 18:05:54.912284 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.912066    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5deb78a3-7dca-485c-801c-7a4188e42108-home\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 18:05:54.939309 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:54.939283    2566 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb") on node "ip-10-0-143-86.ec2.internal"
Jul 27 18:05:55.013464 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:55.013431    2566 reconciler_common.go:299] "Volume detached for volume \"pvc-b2caa7bf-6fe7-483d-a823-08cbb01c9713\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da866186c5a5d3cb\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"csi-630dddfe5bbcc0ec2bfe0a63a4fad07796b940bce0091ad7473a8830e836a5ae\""
Jul 27 18:05:55.178441 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:55.178345    2566 generic.go:356] "Generic (PLEG): container finished" podID="5deb78a3-7dca-485c-801c-7a4188e42108" containerID="ff9ba07fa54244d135e271ef129cdec491f5831e3058797e243ecb05bc3be71a" exitCode=0
Jul 27 18:05:55.178441 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:55.178395    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:05:55.178633 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:55.178434    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"5deb78a3-7dca-485c-801c-7a4188e42108","Type":"ContainerDied","Data":"ff9ba07fa54244d135e271ef129cdec491f5831e3058797e243ecb05bc3be71a"}
Jul 27 18:05:55.178633 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:55.178494    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"5deb78a3-7dca-485c-801c-7a4188e42108","Type":"ContainerDied","Data":"bab03da3436a2577c429ac065a03d56f58ed321cf39ae170bfbc06ca8b8fb186"}
Jul 27 18:05:55.178633 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:55.178517    2566 scope.go:117] "RemoveContainer" containerID="ff9ba07fa54244d135e271ef129cdec491f5831e3058797e243ecb05bc3be71a"
Jul 27 18:05:55.188585 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:55.188559    2566 scope.go:117] "RemoveContainer" containerID="ff9ba07fa54244d135e271ef129cdec491f5831e3058797e243ecb05bc3be71a"
Jul 27 18:05:55.188974 ip-10-0-143-86 kubenswrapper[2566]: E0727 18:05:55.188952    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff9ba07fa54244d135e271ef129cdec491f5831e3058797e243ecb05bc3be71a\": container with ID starting with ff9ba07fa54244d135e271ef129cdec491f5831e3058797e243ecb05bc3be71a not found: ID does not exist" containerID="ff9ba07fa54244d135e271ef129cdec491f5831e3058797e243ecb05bc3be71a"
Jul 27 18:05:55.189056 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:55.188983    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff9ba07fa54244d135e271ef129cdec491f5831e3058797e243ecb05bc3be71a"} err="failed to get container status \"ff9ba07fa54244d135e271ef129cdec491f5831e3058797e243ecb05bc3be71a\": rpc error: code = NotFound desc = could not find container \"ff9ba07fa54244d135e271ef129cdec491f5831e3058797e243ecb05bc3be71a\": container with ID starting with ff9ba07fa54244d135e271ef129cdec491f5831e3058797e243ecb05bc3be71a not found: ID does not exist"
Jul 27 18:05:55.205747 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:55.205710    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 27 18:05:55.213317 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:55.213291    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 27 18:05:56.403179 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:56.403135    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5deb78a3-7dca-485c-801c-7a4188e42108" path="/var/lib/kubelet/pods/5deb78a3-7dca-485c-801c-7a4188e42108/volumes"
Jul 27 18:05:57.762722 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:57.762595    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 18:05:57.779710 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:05:57.779689    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 18:06:04.058179 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:06:04.058113    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599"]
Jul 27 18:06:04.060373 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:06:04.060349    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-ts599"]
Jul 27 18:06:04.404992 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:06:04.404912    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b5d612c-35d6-417e-82d9-3b9aacaae5c6" path="/var/lib/kubelet/pods/0b5d612c-35d6-417e-82d9-3b9aacaae5c6/volumes"
Jul 27 18:07:52.800221 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.800186    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"]
Jul 27 18:07:52.800734 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.800575    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5deb78a3-7dca-485c-801c-7a4188e42108" containerName="main"
Jul 27 18:07:52.800734 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.800588    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5deb78a3-7dca-485c-801c-7a4188e42108" containerName="main"
Jul 27 18:07:52.800734 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.800662    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5deb78a3-7dca-485c-801c-7a4188e42108" containerName="main"
Jul 27 18:07:52.803850 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.803833    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.806951 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.806925    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 27 18:07:52.806951 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.806943    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-npjbj\""
Jul 27 18:07:52.806951 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.806953    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 27 18:07:52.807216 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.806926    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 27 18:07:52.807216 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.807006    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 27 18:07:52.826852 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.826823    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"]
Jul 27 18:07:52.886582 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.886543    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl2bh\" (UniqueName: \"kubernetes.io/projected/8fda2b07-1ccd-4338-bb06-669e34cb31f0-kube-api-access-nl2bh\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.886768 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.886592    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-dshm\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.886768 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.886634    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-model-cache\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.886768 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.886715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fda2b07-1ccd-4338-bb06-669e34cb31f0-tls-certs\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.886768 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.886756    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-tmp-dir\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.886916 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.886777    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-kserve-provision-location\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.886916 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.886843    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-home\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.987500 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.987459    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-model-cache\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.987724 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.987527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fda2b07-1ccd-4338-bb06-669e34cb31f0-tls-certs\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.987724 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.987564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-tmp-dir\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.987724 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.987585    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-kserve-provision-location\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.987724 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.987624    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-home\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.987724 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.987692    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2bh\" (UniqueName: \"kubernetes.io/projected/8fda2b07-1ccd-4338-bb06-669e34cb31f0-kube-api-access-nl2bh\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.987724 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.987723    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-dshm\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.988116 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.988080    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-tmp-dir\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.988285 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.988193    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-kserve-provision-location\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.988370 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.988346    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-model-cache\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.988469 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.988452    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-home\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.990123 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.990093    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-dshm\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.990241 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.990222    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fda2b07-1ccd-4338-bb06-669e34cb31f0-tls-certs\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:52.996800 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:52.996779    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl2bh\" (UniqueName: \"kubernetes.io/projected/8fda2b07-1ccd-4338-bb06-669e34cb31f0-kube-api-access-nl2bh\") pod \"stop-feature-test-kserve-b48b67d59-9kkbb\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:53.114906 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:53.114815    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:07:53.264796 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:53.264760    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"]
Jul 27 18:07:53.265990 ip-10-0-143-86 kubenswrapper[2566]: W0727 18:07:53.265955    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fda2b07_1ccd_4338_bb06_669e34cb31f0.slice/crio-22083cc8a872dc26ba82db30a3ae93cb5f84926f2e0a1e5307087749cd226cd5 WatchSource:0}: Error finding container 22083cc8a872dc26ba82db30a3ae93cb5f84926f2e0a1e5307087749cd226cd5: Status 404 returned error can't find the container with id 22083cc8a872dc26ba82db30a3ae93cb5f84926f2e0a1e5307087749cd226cd5
Jul 27 18:07:53.848278 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:53.848237    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" event={"ID":"8fda2b07-1ccd-4338-bb06-669e34cb31f0","Type":"ContainerStarted","Data":"402af157f9ed9f3e02a8329f136933141f96455153582df102939390fb0f3704"}
Jul 27 18:07:53.848677 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:53.848314    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" event={"ID":"8fda2b07-1ccd-4338-bb06-669e34cb31f0","Type":"ContainerStarted","Data":"22083cc8a872dc26ba82db30a3ae93cb5f84926f2e0a1e5307087749cd226cd5"}
Jul 27 18:07:57.047090 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:07:57.047058    2566 scope.go:117] "RemoveContainer" containerID="90166070deccbe004653d823a62f4ba5df43db8a63bf88525f7f209b98055131"
Jul 27 18:08:01.893755 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:08:01.893720    2566 generic.go:356] "Generic (PLEG): container finished" podID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerID="402af157f9ed9f3e02a8329f136933141f96455153582df102939390fb0f3704" exitCode=0
Jul 27 18:08:01.893755 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:08:01.893760    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" event={"ID":"8fda2b07-1ccd-4338-bb06-669e34cb31f0","Type":"ContainerDied","Data":"402af157f9ed9f3e02a8329f136933141f96455153582df102939390fb0f3704"}
Jul 27 18:08:02.900662 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:08:02.900627    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" event={"ID":"8fda2b07-1ccd-4338-bb06-669e34cb31f0","Type":"ContainerStarted","Data":"515938729ae33c80f185a12b13335b98e6c78a98a124b0f52a670f20a5fb99bf"}
Jul 27 18:08:02.943137 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:08:02.943080    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" podStartSLOduration=3.135330818 podStartE2EDuration="10.943062161s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.807731343s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:08:02.940935137 +0000 UTC m=+4027.016199637" watchObservedRunningTime="2026-07-27 18:08:02.943062161 +0000 UTC m=+4027.018326647"
Jul 27 18:08:03.115407 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:08:03.115315    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:08:03.115407 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:08:03.115397    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:08:03.116955 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:08:03.116915    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 27 18:08:13.116035 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:08:13.115988    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 27 18:08:23.115604 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:08:23.115516    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 27 18:08:33.115490 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:08:33.115447    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 27 18:08:43.116399 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:08:43.116341    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 27 18:08:53.116187 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:08:53.116130    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 27 18:09:03.115396 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:09:03.115290    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 27 18:09:13.116205 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:09:13.116132    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 27 18:09:23.116276 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:09:23.116229    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 27 18:09:33.116085 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:09:33.116037    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 27 18:09:43.115720 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:09:43.115672    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 27 18:09:53.115648 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:09:53.115604    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 27 18:10:03.126371 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:03.126288    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:10:03.134484 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:03.134456    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:10:21.496898 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:21.496865    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"]
Jul 27 18:10:36.519427 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:36.519359    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerName="main" containerID="cri-o://515938729ae33c80f185a12b13335b98e6c78a98a124b0f52a670f20a5fb99bf" gracePeriod=45
Jul 27 18:10:37.127441 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.127401    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 27 18:10:37.282034 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.282007    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:10:37.371426 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.371334    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl2bh\" (UniqueName: \"kubernetes.io/projected/8fda2b07-1ccd-4338-bb06-669e34cb31f0-kube-api-access-nl2bh\") pod \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") "
Jul 27 18:10:37.371598 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.371431    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-tmp-dir\") pod \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") "
Jul 27 18:10:37.371598 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.371492    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-home\") pod \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") "
Jul 27 18:10:37.371598 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.371516    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-dshm\") pod \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") "
Jul 27 18:10:37.371598 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.371540    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-kserve-provision-location\") pod \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") "
Jul 27 18:10:37.371598 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.371568    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fda2b07-1ccd-4338-bb06-669e34cb31f0-tls-certs\") pod \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") "
Jul 27 18:10:37.371856 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.371607    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-model-cache\") pod \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\" (UID: \"8fda2b07-1ccd-4338-bb06-669e34cb31f0\") "
Jul 27 18:10:37.372031 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.372000    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "8fda2b07-1ccd-4338-bb06-669e34cb31f0" (UID: "8fda2b07-1ccd-4338-bb06-669e34cb31f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:10:37.372204 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.372163    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-home" (OuterVolumeSpecName: "home") pod "8fda2b07-1ccd-4338-bb06-669e34cb31f0" (UID: "8fda2b07-1ccd-4338-bb06-669e34cb31f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:10:37.374342 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.374313    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-dshm" (OuterVolumeSpecName: "dshm") pod "8fda2b07-1ccd-4338-bb06-669e34cb31f0" (UID: "8fda2b07-1ccd-4338-bb06-669e34cb31f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:10:37.374479 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.374461    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fda2b07-1ccd-4338-bb06-669e34cb31f0-kube-api-access-nl2bh" (OuterVolumeSpecName: "kube-api-access-nl2bh") pod "8fda2b07-1ccd-4338-bb06-669e34cb31f0" (UID: "8fda2b07-1ccd-4338-bb06-669e34cb31f0"). InnerVolumeSpecName "kube-api-access-nl2bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:10:37.374544 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.374461    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fda2b07-1ccd-4338-bb06-669e34cb31f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fda2b07-1ccd-4338-bb06-669e34cb31f0" (UID: "8fda2b07-1ccd-4338-bb06-669e34cb31f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:10:37.385364 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.385336    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8fda2b07-1ccd-4338-bb06-669e34cb31f0" (UID: "8fda2b07-1ccd-4338-bb06-669e34cb31f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:10:37.413515 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.413478    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8fda2b07-1ccd-4338-bb06-669e34cb31f0" (UID: "8fda2b07-1ccd-4338-bb06-669e34cb31f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:10:37.473495 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.473455    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-tmp-dir\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 18:10:37.473495 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.473495    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-home\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 18:10:37.473747 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.473508    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-dshm\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 18:10:37.473747 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.473524    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-kserve-provision-location\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 18:10:37.473747 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.473541    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fda2b07-1ccd-4338-bb06-669e34cb31f0-tls-certs\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 18:10:37.473747 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.473555    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fda2b07-1ccd-4338-bb06-669e34cb31f0-model-cache\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 18:10:37.473747 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.473569    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl2bh\" (UniqueName: \"kubernetes.io/projected/8fda2b07-1ccd-4338-bb06-669e34cb31f0-kube-api-access-nl2bh\") on node \"ip-10-0-143-86.ec2.internal\" DevicePath \"\""
Jul 27 18:10:37.842116 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.842085    2566 generic.go:356] "Generic (PLEG): container finished" podID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerID="515938729ae33c80f185a12b13335b98e6c78a98a124b0f52a670f20a5fb99bf" exitCode=0
Jul 27 18:10:37.842531 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.842138    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"
Jul 27 18:10:37.842531 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.842158    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" event={"ID":"8fda2b07-1ccd-4338-bb06-669e34cb31f0","Type":"ContainerDied","Data":"515938729ae33c80f185a12b13335b98e6c78a98a124b0f52a670f20a5fb99bf"}
Jul 27 18:10:37.842531 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.842215    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb" event={"ID":"8fda2b07-1ccd-4338-bb06-669e34cb31f0","Type":"ContainerDied","Data":"22083cc8a872dc26ba82db30a3ae93cb5f84926f2e0a1e5307087749cd226cd5"}
Jul 27 18:10:37.842531 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.842235    2566 scope.go:117] "RemoveContainer" containerID="515938729ae33c80f185a12b13335b98e6c78a98a124b0f52a670f20a5fb99bf"
Jul 27 18:10:37.851705 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.851683    2566 scope.go:117] "RemoveContainer" containerID="402af157f9ed9f3e02a8329f136933141f96455153582df102939390fb0f3704"
Jul 27 18:10:37.860102 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.860086    2566 scope.go:117] "RemoveContainer" containerID="515938729ae33c80f185a12b13335b98e6c78a98a124b0f52a670f20a5fb99bf"
Jul 27 18:10:37.860417 ip-10-0-143-86 kubenswrapper[2566]: E0727 18:10:37.860396    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515938729ae33c80f185a12b13335b98e6c78a98a124b0f52a670f20a5fb99bf\": container with ID starting with 515938729ae33c80f185a12b13335b98e6c78a98a124b0f52a670f20a5fb99bf not found: ID does not exist" containerID="515938729ae33c80f185a12b13335b98e6c78a98a124b0f52a670f20a5fb99bf"
Jul 27 18:10:37.860468 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.860425    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515938729ae33c80f185a12b13335b98e6c78a98a124b0f52a670f20a5fb99bf"} err="failed to get container status \"515938729ae33c80f185a12b13335b98e6c78a98a124b0f52a670f20a5fb99bf\": rpc error: code = NotFound desc = could not find container \"515938729ae33c80f185a12b13335b98e6c78a98a124b0f52a670f20a5fb99bf\": container with ID starting with 515938729ae33c80f185a12b13335b98e6c78a98a124b0f52a670f20a5fb99bf not found: ID does not exist"
Jul 27 18:10:37.860468 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.860442    2566 scope.go:117] "RemoveContainer" containerID="402af157f9ed9f3e02a8329f136933141f96455153582df102939390fb0f3704"
Jul 27 18:10:37.860713 ip-10-0-143-86 kubenswrapper[2566]: E0727 18:10:37.860696    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"402af157f9ed9f3e02a8329f136933141f96455153582df102939390fb0f3704\": container with ID starting with 402af157f9ed9f3e02a8329f136933141f96455153582df102939390fb0f3704 not found: ID does not exist" containerID="402af157f9ed9f3e02a8329f136933141f96455153582df102939390fb0f3704"
Jul 27 18:10:37.860768 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.860719    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"402af157f9ed9f3e02a8329f136933141f96455153582df102939390fb0f3704"} err="failed to get container status \"402af157f9ed9f3e02a8329f136933141f96455153582df102939390fb0f3704\": rpc error: code = NotFound desc = could not find container \"402af157f9ed9f3e02a8329f136933141f96455153582df102939390fb0f3704\": container with ID starting with 402af157f9ed9f3e02a8329f136933141f96455153582df102939390fb0f3704 not found: ID does not exist"
Jul 27 18:10:37.869368 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.869342    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"]
Jul 27 18:10:37.873388 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:37.873365    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-9kkbb"]
Jul 27 18:10:38.403365 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:38.403327    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" path="/var/lib/kubelet/pods/8fda2b07-1ccd-4338-bb06-669e34cb31f0/volumes"
Jul 27 18:10:39.519722 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.519685    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"]
Jul 27 18:10:39.520097 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.520067    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerName="storage-initializer"
Jul 27 18:10:39.520097 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.520079    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerName="storage-initializer"
Jul 27 18:10:39.520097 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.520092    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerName="main"
Jul 27 18:10:39.520097 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.520098    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerName="main"
Jul 27 18:10:39.520283 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.520151    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fda2b07-1ccd-4338-bb06-669e34cb31f0" containerName="main"
Jul 27 18:10:39.525232 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.525203    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.528134 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.528107    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-npjbj\""
Jul 27 18:10:39.528427 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.528406    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 27 18:10:39.528525 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.528455    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 27 18:10:39.528586 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.528546    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 27 18:10:39.529089 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.529069    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 27 18:10:39.540002 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.539980    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"]
Jul 27 18:10:39.697927 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.697890    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-home\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.697927 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.697942    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-tmp-dir\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.698240 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.698030    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-tls-certs\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.698240 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.698074    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-dshm\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.698240 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.698100    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-kserve-provision-location\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.698240 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.698154    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-model-cache\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.698448 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.698252    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbkl2\" (UniqueName: \"kubernetes.io/projected/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-kube-api-access-nbkl2\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.798866 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.798766    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-tls-certs\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.798866 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.798806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-dshm\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.798866 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.798828    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-kserve-provision-location\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.799161 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.798871    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-model-cache\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.799161 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.798900    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbkl2\" (UniqueName: \"kubernetes.io/projected/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-kube-api-access-nbkl2\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.799161 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.798941    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-home\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.799161 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.798987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-tmp-dir\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.799375 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.799333    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-kserve-provision-location\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.799486 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.799463    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-model-cache\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.799534 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.799467    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-tmp-dir\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.799534 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.799522    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-home\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.801076 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.801054    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-dshm\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.801534 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.801513    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-tls-certs\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.809594 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.809558    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbkl2\" (UniqueName: \"kubernetes.io/projected/5f1158cf-5c31-47bf-ba3e-5b54c3571c46-kube-api-access-nbkl2\") pod \"stop-feature-test-kserve-b48b67d59-8fj65\" (UID: \"5f1158cf-5c31-47bf-ba3e-5b54c3571c46\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.835976 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.835950    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:39.987631 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:39.987590    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"]
Jul 27 18:10:39.990481 ip-10-0-143-86 kubenswrapper[2566]: W0727 18:10:39.990435    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1158cf_5c31_47bf_ba3e_5b54c3571c46.slice/crio-c3d17b31da3307f282771b8de8e8e5e98c49bd93ab7940b845d70b70b7b79dc2 WatchSource:0}: Error finding container c3d17b31da3307f282771b8de8e8e5e98c49bd93ab7940b845d70b70b7b79dc2: Status 404 returned error can't find the container with id c3d17b31da3307f282771b8de8e8e5e98c49bd93ab7940b845d70b70b7b79dc2
Jul 27 18:10:40.861985 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:40.861953    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65" event={"ID":"5f1158cf-5c31-47bf-ba3e-5b54c3571c46","Type":"ContainerStarted","Data":"d7239af1098296150c05426982cba5dcaf55e9994c7970e2e78ea84464a40dd5"}
Jul 27 18:10:40.862369 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:40.862005    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65" event={"ID":"5f1158cf-5c31-47bf-ba3e-5b54c3571c46","Type":"ContainerStarted","Data":"c3d17b31da3307f282771b8de8e8e5e98c49bd93ab7940b845d70b70b7b79dc2"}
Jul 27 18:10:47.909392 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:47.909349    2566 generic.go:356] "Generic (PLEG): container finished" podID="5f1158cf-5c31-47bf-ba3e-5b54c3571c46" containerID="d7239af1098296150c05426982cba5dcaf55e9994c7970e2e78ea84464a40dd5" exitCode=0
Jul 27 18:10:47.909799 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:47.909429    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65" event={"ID":"5f1158cf-5c31-47bf-ba3e-5b54c3571c46","Type":"ContainerDied","Data":"d7239af1098296150c05426982cba5dcaf55e9994c7970e2e78ea84464a40dd5"}
Jul 27 18:10:48.916741 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:48.916705    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65" event={"ID":"5f1158cf-5c31-47bf-ba3e-5b54c3571c46","Type":"ContainerStarted","Data":"14a78463ec1bcf4e0ede5344fa2529b1c26ef40ef50741fa48ef1b866944d39a"}
Jul 27 18:10:48.943691 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:48.943636    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65" podStartSLOduration=2.304429873 podStartE2EDuration="9.943621172s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.639191299s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:10:48.940134794 +0000 UTC m=+4193.015399286" watchObservedRunningTime="2026-07-27 18:10:48.943621172 +0000 UTC m=+4193.018885658"
Jul 27 18:10:49.836352 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:49.836313    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:49.836568 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:49.836425    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:10:49.838795 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:49.838758    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65" podUID="5f1158cf-5c31-47bf-ba3e-5b54c3571c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 27 18:10:57.804063 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:57.803946    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 18:10:57.824409 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:57.824384    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 18:10:59.837060 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:10:59.837019    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65" podUID="5f1158cf-5c31-47bf-ba3e-5b54c3571c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 27 18:11:09.836263 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:11:09.836221    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65" podUID="5f1158cf-5c31-47bf-ba3e-5b54c3571c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 27 18:11:19.837533 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:11:19.837432    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65" podUID="5f1158cf-5c31-47bf-ba3e-5b54c3571c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 27 18:11:29.837150 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:11:29.837108    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65" podUID="5f1158cf-5c31-47bf-ba3e-5b54c3571c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 27 18:11:39.837550 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:11:39.837506    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65" podUID="5f1158cf-5c31-47bf-ba3e-5b54c3571c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 27 18:11:49.837051 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:11:49.836994    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65" podUID="5f1158cf-5c31-47bf-ba3e-5b54c3571c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 27 18:11:59.837181 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:11:59.837116    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65" podUID="5f1158cf-5c31-47bf-ba3e-5b54c3571c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 27 18:12:09.416397 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:09.416301    2566 generic.go:356] "Generic (PLEG): container finished" podID="51a5c4f1-8809-446a-90e5-a987d68ffb14" containerID="a040a617960e1d020c92d028caf985a1690fe70ec27c228ab7d37d522888210e" exitCode=1
Jul 27 18:12:09.416397 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:09.416372    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28" event={"ID":"51a5c4f1-8809-446a-90e5-a987d68ffb14","Type":"ContainerDied","Data":"a040a617960e1d020c92d028caf985a1690fe70ec27c228ab7d37d522888210e"}
Jul 27 18:12:09.416909 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:09.416757    2566 scope.go:117] "RemoveContainer" containerID="a040a617960e1d020c92d028caf985a1690fe70ec27c228ab7d37d522888210e"
Jul 27 18:12:09.417188 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:09.417152    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 18:12:09.418182 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:09.418086    2566 generic.go:356] "Generic (PLEG): container finished" podID="32959584-b9bb-4d73-92ad-5433f7c5f44b" containerID="4b4d3b820cbd1ca5714a22dc7bf2e4de5a6b29e15a1fa7600fc61b4872973286" exitCode=1
Jul 27 18:12:09.418246 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:09.418187    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw" event={"ID":"32959584-b9bb-4d73-92ad-5433f7c5f44b","Type":"ContainerDied","Data":"4b4d3b820cbd1ca5714a22dc7bf2e4de5a6b29e15a1fa7600fc61b4872973286"}
Jul 27 18:12:09.418540 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:09.418525    2566 scope.go:117] "RemoveContainer" containerID="4b4d3b820cbd1ca5714a22dc7bf2e4de5a6b29e15a1fa7600fc61b4872973286"
Jul 27 18:12:09.837139 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:09.837089    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65" podUID="5f1158cf-5c31-47bf-ba3e-5b54c3571c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 27 18:12:11.433057 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:11.433017    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28" event={"ID":"51a5c4f1-8809-446a-90e5-a987d68ffb14","Type":"ContainerStarted","Data":"0d55c0325ace8a900c5a8914fd0bb1aaee54685b9ca6a2fd7184c573c9d38eb0"}
Jul 27 18:12:11.433541 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:11.433318    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28"
Jul 27 18:12:11.435468 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:11.435436    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw" event={"ID":"32959584-b9bb-4d73-92ad-5433f7c5f44b","Type":"ContainerStarted","Data":"5fd752417b4fb00c9d05f5ca4ae6d920df94b34b019a3f426701a4cde88a7eff"}
Jul 27 18:12:11.435645 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:11.435626    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw"
Jul 27 18:12:19.837293 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:19.837224    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65" podUID="5f1158cf-5c31-47bf-ba3e-5b54c3571c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 27 18:12:22.503399 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:22.503362    2566 generic.go:356] "Generic (PLEG): container finished" podID="32959584-b9bb-4d73-92ad-5433f7c5f44b" containerID="5fd752417b4fb00c9d05f5ca4ae6d920df94b34b019a3f426701a4cde88a7eff" exitCode=1
Jul 27 18:12:22.503863 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:22.503417    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw" event={"ID":"32959584-b9bb-4d73-92ad-5433f7c5f44b","Type":"ContainerDied","Data":"5fd752417b4fb00c9d05f5ca4ae6d920df94b34b019a3f426701a4cde88a7eff"}
Jul 27 18:12:22.503863 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:22.503476    2566 scope.go:117] "RemoveContainer" containerID="4b4d3b820cbd1ca5714a22dc7bf2e4de5a6b29e15a1fa7600fc61b4872973286"
Jul 27 18:12:22.503863 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:22.503841    2566 scope.go:117] "RemoveContainer" containerID="5fd752417b4fb00c9d05f5ca4ae6d920df94b34b019a3f426701a4cde88a7eff"
Jul 27 18:12:22.504133 ip-10-0-143-86 kubenswrapper[2566]: E0727 18:12:22.504104    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-77cb7f7d68-fl4nw_kserve(32959584-b9bb-4d73-92ad-5433f7c5f44b)\"" pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw" podUID="32959584-b9bb-4d73-92ad-5433f7c5f44b"
Jul 27 18:12:27.886267 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:27.886231    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw"
Jul 27 18:12:27.886663 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:27.886606    2566 scope.go:117] "RemoveContainer" containerID="5fd752417b4fb00c9d05f5ca4ae6d920df94b34b019a3f426701a4cde88a7eff"
Jul 27 18:12:27.886799 ip-10-0-143-86 kubenswrapper[2566]: E0727 18:12:27.886778    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-77cb7f7d68-fl4nw_kserve(32959584-b9bb-4d73-92ad-5433f7c5f44b)\"" pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw" podUID="32959584-b9bb-4d73-92ad-5433f7c5f44b"
Jul 27 18:12:29.836613 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:29.836565    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65" podUID="5f1158cf-5c31-47bf-ba3e-5b54c3571c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 27 18:12:32.443608 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:32.443574    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-d7z28"
Jul 27 18:12:39.837093 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:39.837044    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65" podUID="5f1158cf-5c31-47bf-ba3e-5b54c3571c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 27 18:12:42.397361 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:42.397326    2566 scope.go:117] "RemoveContainer" containerID="5fd752417b4fb00c9d05f5ca4ae6d920df94b34b019a3f426701a4cde88a7eff"
Jul 27 18:12:43.638349 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:43.638311    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw" event={"ID":"32959584-b9bb-4d73-92ad-5433f7c5f44b","Type":"ContainerStarted","Data":"04223f814adfa5c37f1661b6bcc5a41f9c4d621b85e2a3c7199e47e53c263367"}
Jul 27 18:12:43.638734 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:43.638515    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw"
Jul 27 18:12:49.846569 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:49.846495    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:12:49.854185 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:12:49.854143    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-b48b67d59-8fj65"
Jul 27 18:13:14.645329 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:13:14.645287    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-77cb7f7d68-fl4nw"
Jul 27 18:15:57.846587 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:15:57.846458    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 18:15:57.868992 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:15:57.868966    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 18:20:57.889129 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:20:57.889009    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 18:20:57.909543 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:20:57.909517    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 18:25:57.937559 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:25:57.937447    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 18:25:57.951220 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:25:57.951190    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 18:29:46.946693 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:29:46.946598    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c_5854c1ae-85a1-4d4f-8d45-792d72188f40/istio-proxy/0.log"
Jul 27 18:29:47.002652 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:29:47.002619    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c_5854c1ae-85a1-4d4f-8d45-792d72188f40/istio-proxy/1.log"
Jul 27 18:29:48.133668 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:29:48.133636    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c_5854c1ae-85a1-4d4f-8d45-792d72188f40/istio-proxy/0.log"
Jul 27 18:29:48.181287 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:29:48.181250    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c_5854c1ae-85a1-4d4f-8d45-792d72188f40/istio-proxy/1.log"
Jul 27 18:29:49.384865 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:29:49.384830    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c_5854c1ae-85a1-4d4f-8d45-792d72188f40/istio-proxy/0.log"
Jul 27 18:29:49.431902 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:29:49.431865    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c_5854c1ae-85a1-4d4f-8d45-792d72188f40/istio-proxy/1.log"
Jul 27 18:29:50.579494 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:29:50.579460    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c_5854c1ae-85a1-4d4f-8d45-792d72188f40/istio-proxy/0.log"
Jul 27 18:29:50.624790 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:29:50.624761    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c_5854c1ae-85a1-4d4f-8d45-792d72188f40/istio-proxy/1.log"
Jul 27 18:29:51.590179 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:29:51.590129    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:29:52.880617 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:29:52.880579    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:29:54.208275 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:29:54.208239    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:29:55.426176 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:29:55.426122    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:29:56.755571 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:29:56.755537    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:29:57.986008 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:29:57.985975    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:29:59.227368 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:29:59.227335    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:00.482333 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:00.482296    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:01.754186 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:01.754141    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:03.171821 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:03.171785    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:04.487719 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:04.487682    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:05.775920 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:05.775881    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:07.044015 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:07.043977    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:08.312721 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:08.312688    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:09.558384 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:09.558349    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:10.753832 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:10.753791    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:11.986585 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:11.986550    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:13.221550 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:13.221517    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:14.518820 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:14.518782    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:15.768008 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:15.767972    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:17.010768 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:17.010735    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:18.224183 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:18.224125    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:19.499082 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:19.499040    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:20.762594 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:20.762563    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:22.029467 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:22.029425    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:23.323604 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:23.323571    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:24.582495 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:24.582461    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:25.846483 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:25.846454    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:27.110866 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:27.110825    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:28.367286 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:28.367251    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:29.658366 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:29.658331    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:30.946953 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:30.946916    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:32.244237 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:32.244193    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:33.539011 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:33.538972    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:34.806507 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:34.806429    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:36.100914 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:36.100877    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:37.406733 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:37.406699    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:38.672615 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:38.672573    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:39.930597 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:39.930557    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:41.187597 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:41.187559    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:42.472622 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:42.472590    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:43.742944 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:43.742908    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:45.027902 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:45.027850    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:46.297932 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:46.297895    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:47.594725 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:47.594693    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:48.854663 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:48.854573    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:50.115561 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:50.115525    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:51.367224 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:51.367188    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:52.651556 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:52.651523    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:30:57.977761 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:57.977728    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 18:30:57.992021 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:57.991994    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m9k9v_1b606708-d7b9-4eab-8da8-ecfd50f006fb/ovn-acl-logging/0.log"
Jul 27 18:30:58.329850 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:58.329749    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8jgtw_16bb5aa2-183e-4742-a424-4ba30c7e3d95/global-pull-secret-syncer/0.log"
Jul 27 18:30:58.494634 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:58.494585    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vclxn_2a5e96ae-f5a4-4e64-9c29-1dbcb1f9cc4d/konnectivity-agent/0.log"
Jul 27 18:30:58.647212 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:30:58.647114    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-86.ec2.internal_81d1b794894ec66f164d640b4bfc3b9a/haproxy/0.log"
Jul 27 18:31:03.367690 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:03.367650    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-s6nzs_107c0c61-f4e0-4f44-b2e1-5ad3beaccfdc/authorino/0.log"
Jul 27 18:31:05.046232 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:05.046202    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e906720c-c8d1-4be6-9f5f-82bf93864fb0/alertmanager/0.log"
Jul 27 18:31:05.075995 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:05.075908    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e906720c-c8d1-4be6-9f5f-82bf93864fb0/config-reloader/0.log"
Jul 27 18:31:05.100507 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:05.100480    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e906720c-c8d1-4be6-9f5f-82bf93864fb0/kube-rbac-proxy-web/0.log"
Jul 27 18:31:05.124732 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:05.124707    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e906720c-c8d1-4be6-9f5f-82bf93864fb0/kube-rbac-proxy/0.log"
Jul 27 18:31:05.149815 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:05.149786    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e906720c-c8d1-4be6-9f5f-82bf93864fb0/kube-rbac-proxy-metric/0.log"
Jul 27 18:31:05.173839 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:05.173811    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e906720c-c8d1-4be6-9f5f-82bf93864fb0/prom-label-proxy/0.log"
Jul 27 18:31:05.199090 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:05.199063    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e906720c-c8d1-4be6-9f5f-82bf93864fb0/init-config-reloader/0.log"
Jul 27 18:31:05.265134 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:05.265100    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-dzpt4_50804eac-c593-411c-9aaa-f4a3742cb1fe/cluster-monitoring-operator/0.log"
Jul 27 18:31:05.551617 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:05.551584    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7m5zw_48f4573e-95be-44cb-b1e4-6773d04d7f68/node-exporter/0.log"
Jul 27 18:31:05.578467 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:05.578439    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7m5zw_48f4573e-95be-44cb-b1e4-6773d04d7f68/kube-rbac-proxy/0.log"
Jul 27 18:31:05.611863 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:05.611836    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7m5zw_48f4573e-95be-44cb-b1e4-6773d04d7f68/init-textfile/0.log"
Jul 27 18:31:05.853001 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:05.852928    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-dgxj6_ff30bbdb-a210-458d-b79e-be33cd423407/kube-rbac-proxy-main/0.log"
Jul 27 18:31:05.893185 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:05.893130    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-dgxj6_ff30bbdb-a210-458d-b79e-be33cd423407/kube-rbac-proxy-self/0.log"
Jul 27 18:31:05.929443 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:05.929406    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-dgxj6_ff30bbdb-a210-458d-b79e-be33cd423407/openshift-state-metrics/0.log"
Jul 27 18:31:06.187321 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:06.187235    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-p4prc_d663fddb-ec5a-48ed-907b-ab957759a948/prometheus-operator/0.log"
Jul 27 18:31:06.210465 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:06.210440    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-p4prc_d663fddb-ec5a-48ed-907b-ab957759a948/kube-rbac-proxy/0.log"
Jul 27 18:31:06.244488 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:06.244460    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-mgf8g_b9850d85-2dad-44dc-83a3-4edf23d796a5/prometheus-operator-admission-webhook/0.log"
Jul 27 18:31:06.272396 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:06.272370    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56ff85d9f4-q8crh_9d51eefb-51d0-42fc-83f9-249fd32b7ba5/telemeter-client/0.log"
Jul 27 18:31:06.296854 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:06.296821    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56ff85d9f4-q8crh_9d51eefb-51d0-42fc-83f9-249fd32b7ba5/reload/0.log"
Jul 27 18:31:06.321009 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:06.320980    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56ff85d9f4-q8crh_9d51eefb-51d0-42fc-83f9-249fd32b7ba5/kube-rbac-proxy/0.log"
Jul 27 18:31:06.355491 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:06.355456    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc46f647f-mr6gc_04058bb3-5ad0-4102-82c5-60e7d32d445c/thanos-query/0.log"
Jul 27 18:31:06.378131 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:06.378101    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc46f647f-mr6gc_04058bb3-5ad0-4102-82c5-60e7d32d445c/kube-rbac-proxy-web/0.log"
Jul 27 18:31:06.400828 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:06.400788    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc46f647f-mr6gc_04058bb3-5ad0-4102-82c5-60e7d32d445c/kube-rbac-proxy/0.log"
Jul 27 18:31:06.427933 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:06.427902    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc46f647f-mr6gc_04058bb3-5ad0-4102-82c5-60e7d32d445c/prom-label-proxy/0.log"
Jul 27 18:31:06.452816 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:06.452785    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc46f647f-mr6gc_04058bb3-5ad0-4102-82c5-60e7d32d445c/kube-rbac-proxy-rules/0.log"
Jul 27 18:31:06.477590 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:06.477557    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dc46f647f-mr6gc_04058bb3-5ad0-4102-82c5-60e7d32d445c/kube-rbac-proxy-metrics/0.log"
Jul 27 18:31:08.889075 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:08.889037    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"]
Jul 27 18:31:08.892667 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:08.892646    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"
Jul 27 18:31:08.894999 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:08.894975    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q5775\"/\"kube-root-ca.crt\""
Jul 27 18:31:08.895894 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:08.895874    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q5775\"/\"default-dockercfg-gsljn\""
Jul 27 18:31:08.895957 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:08.895909    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q5775\"/\"openshift-service-ca.crt\""
Jul 27 18:31:08.901760 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:08.901728    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"]
Jul 27 18:31:08.937447 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:08.937407    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f-lib-modules\") pod \"perf-node-gather-daemonset-8rtcm\" (UID: \"1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"
Jul 27 18:31:08.937447 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:08.937450    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l66xg\" (UniqueName: \"kubernetes.io/projected/1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f-kube-api-access-l66xg\") pod \"perf-node-gather-daemonset-8rtcm\" (UID: \"1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"
Jul 27 18:31:08.937673 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:08.937524    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f-podres\") pod \"perf-node-gather-daemonset-8rtcm\" (UID: \"1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"
Jul 27 18:31:08.937673 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:08.937580    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f-proc\") pod \"perf-node-gather-daemonset-8rtcm\" (UID: \"1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"
Jul 27 18:31:08.937673 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:08.937601    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f-sys\") pod \"perf-node-gather-daemonset-8rtcm\" (UID: \"1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"
Jul 27 18:31:09.038518 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:09.038477    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f-lib-modules\") pod \"perf-node-gather-daemonset-8rtcm\" (UID: \"1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"
Jul 27 18:31:09.038701 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:09.038530    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l66xg\" (UniqueName: \"kubernetes.io/projected/1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f-kube-api-access-l66xg\") pod \"perf-node-gather-daemonset-8rtcm\" (UID: \"1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"
Jul 27 18:31:09.038701 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:09.038584    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f-podres\") pod \"perf-node-gather-daemonset-8rtcm\" (UID: \"1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"
Jul 27 18:31:09.038701 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:09.038622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f-proc\") pod \"perf-node-gather-daemonset-8rtcm\" (UID: \"1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"
Jul 27 18:31:09.038701 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:09.038644    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f-sys\") pod \"perf-node-gather-daemonset-8rtcm\" (UID: \"1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"
Jul 27 18:31:09.038701 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:09.038673    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f-lib-modules\") pod \"perf-node-gather-daemonset-8rtcm\" (UID: \"1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"
Jul 27 18:31:09.038905 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:09.038700    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f-podres\") pod \"perf-node-gather-daemonset-8rtcm\" (UID: \"1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"
Jul 27 18:31:09.038905 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:09.038721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f-sys\") pod \"perf-node-gather-daemonset-8rtcm\" (UID: \"1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"
Jul 27 18:31:09.038905 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:09.038758    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f-proc\") pod \"perf-node-gather-daemonset-8rtcm\" (UID: \"1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"
Jul 27 18:31:09.046600 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:09.046568    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l66xg\" (UniqueName: \"kubernetes.io/projected/1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f-kube-api-access-l66xg\") pod \"perf-node-gather-daemonset-8rtcm\" (UID: \"1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"
Jul 27 18:31:09.203774 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:09.203730    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"
Jul 27 18:31:09.347597 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:09.347562    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"]
Jul 27 18:31:09.922297 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:09.922243    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm" event={"ID":"1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f","Type":"ContainerStarted","Data":"4363976f303a7a28a8df0eed07cac85b06e1dd6e8a59cb6b9543772eea9506a6"}
Jul 27 18:31:09.922753 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:09.922326    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm" event={"ID":"1b59e4a3-664f-4e49-bc3d-28ed3a7fe98f","Type":"ContainerStarted","Data":"1567efcc83719bc53b228c12b896d4faa72f315caf285acef0817264009a21f1"}
Jul 27 18:31:09.922753 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:09.922458    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"
Jul 27 18:31:09.941414 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:09.941359    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm" podStartSLOduration=1.941342781 podStartE2EDuration="1.941342781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:31:09.938255108 +0000 UTC m=+5414.013519596" watchObservedRunningTime="2026-07-27 18:31:09.941342781 +0000 UTC m=+5414.016607268"
Jul 27 18:31:10.490224 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:10.490187    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-prbcj_acde9127-43b5-46ed-9a95-515b25b0faf7/dns/0.log"
Jul 27 18:31:10.514084 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:10.514048    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-prbcj_acde9127-43b5-46ed-9a95-515b25b0faf7/kube-rbac-proxy/0.log"
Jul 27 18:31:10.672492 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:10.672451    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6kcft_e02881b5-7256-40d5-9ad4-199b7c349a48/dns-node-resolver/0.log"
Jul 27 18:31:11.254931 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:11.254898    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2hn7j_676386a3-b937-45a3-88e9-54649e0839e0/node-ca/0.log"
Jul 27 18:31:12.691426 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:12.691394    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c_5854c1ae-85a1-4d4f-8d45-792d72188f40/istio-proxy/0.log"
Jul 27 18:31:12.739162 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:12.739126    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c_5854c1ae-85a1-4d4f-8d45-792d72188f40/istio-proxy/1.log"
Jul 27 18:31:13.254657 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:13.254616    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6tmv2_9d9ff7dd-7fa8-4a73-9b78-76fdd0531ef7/serve-healthcheck-canary/0.log"
Jul 27 18:31:13.789682 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:13.789650    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-2vf8b_9a26877e-9895-4df2-b890-bf52e79a3239/insights-operator/0.log"
Jul 27 18:31:13.791575 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:13.791547    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-2vf8b_9a26877e-9895-4df2-b890-bf52e79a3239/insights-operator/1.log"
Jul 27 18:31:13.888730 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:13.888698    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l6rw6_e299c5a3-e107-464a-ac41-444f951b507c/kube-rbac-proxy/0.log"
Jul 27 18:31:13.911470 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:13.911427    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l6rw6_e299c5a3-e107-464a-ac41-444f951b507c/exporter/0.log"
Jul 27 18:31:13.934915 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:13.934889    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l6rw6_e299c5a3-e107-464a-ac41-444f951b507c/extractor/0.log"
Jul 27 18:31:15.938998 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:15.938964    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q5775/perf-node-gather-daemonset-8rtcm"
Jul 27 18:31:17.073749 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:17.073717    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-58bd97cf4-r6dk4_d9d9b23f-2053-4ec9-a65d-bfa2109dc8b2/manager/0.log"
Jul 27 18:31:17.820561 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:17.820535    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6ccfdc6bb8-tsbd7_6edde0d5-156d-4368-91de-48b54e7b7087/manager/0.log"
Jul 27 18:31:17.846217 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:17.846152    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-77cb7f7d68-fl4nw_32959584-b9bb-4d73-92ad-5433f7c5f44b/manager/1.log"
Jul 27 18:31:17.994033 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:17.993998    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-77cb7f7d68-fl4nw_32959584-b9bb-4d73-92ad-5433f7c5f44b/manager/2.log"
Jul 27 18:31:18.848743 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:18.848653    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9m24t_e66aa8fe-8604-4b2e-bcb8-f171a15c374e/s3-init/0.log"
Jul 27 18:31:18.867830 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:18.867802    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9m24t_e66aa8fe-8604-4b2e-bcb8-f171a15c374e/download-hf-model/0.log"
Jul 27 18:31:19.004532 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:19.004499    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c_5854c1ae-85a1-4d4f-8d45-792d72188f40/istio-proxy/0.log"
Jul 27 18:31:19.051444 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:19.051415    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c_5854c1ae-85a1-4d4f-8d45-792d72188f40/istio-proxy/1.log"
Jul 27 18:31:20.493751 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:20.493721    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c_5854c1ae-85a1-4d4f-8d45-792d72188f40/istio-proxy/0.log"
Jul 27 18:31:20.541236 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:20.541197    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-9jj7c_5854c1ae-85a1-4d4f-8d45-792d72188f40/istio-proxy/1.log"
Jul 27 18:31:25.349137 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:25.349100    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8m5b9_952b4a9c-f2f9-4bf9-a02b-cc426d1995c1/kube-multus/0.log"
Jul 27 18:31:25.375350 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:25.375321    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mjl8_edb57dae-199b-48e9-9a8c-592f3101d307/kube-multus-additional-cni-plugins/0.log"
Jul 27 18:31:25.396775 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:25.396733    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mjl8_edb57dae-199b-48e9-9a8c-592f3101d307/egress-router-binary-copy/0.log"
Jul 27 18:31:25.419468 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:25.419445    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mjl8_edb57dae-199b-48e9-9a8c-592f3101d307/cni-plugins/0.log"
Jul 27 18:31:25.442615 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:25.442597    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mjl8_edb57dae-199b-48e9-9a8c-592f3101d307/bond-cni-plugin/0.log"
Jul 27 18:31:25.465330 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:25.465307    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mjl8_edb57dae-199b-48e9-9a8c-592f3101d307/routeoverride-cni/0.log"
Jul 27 18:31:25.486541 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:25.486517    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mjl8_edb57dae-199b-48e9-9a8c-592f3101d307/whereabouts-cni-bincopy/0.log"
Jul 27 18:31:25.509423 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:25.509398    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2mjl8_edb57dae-199b-48e9-9a8c-592f3101d307/whereabouts-cni/0.log"
Jul 27 18:31:25.858328 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:25.858296    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d72jm_e0651122-b4fd-4a56-8909-53661d776bfc/network-metrics-daemon/0.log"
Jul 27 18:31:25.875799 ip-10-0-143-86 kubenswrapper[2566]: I0727 18:31:25.875776    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d72jm_e0651122-b4fd-4a56-8909-53661d776bfc/kube-rbac-proxy/0.log"