Sep 17 17:38:43.396570 ip-10-0-136-127 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 17:38:43.396581 ip-10-0-136-127 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 17:38:43.396589 ip-10-0-136-127 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 17:38:43.396734 ip-10-0-136-127 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 17:38:53.528696 ip-10-0-136-127 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 17:38:53.528716 ip-10-0-136-127 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 81b1109c47be401c97370c754a238407 --
Sep 17 17:40:58.897857 ip-10-0-136-127 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:41:02.333282 ip-10-0-136-127 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:41:02.333282 ip-10-0-136-127 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:41:02.333282 ip-10-0-136-127 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:41:02.333282 ip-10-0-136-127 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:41:02.333282 ip-10-0-136-127 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:41:02.334098 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.333347    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:41:02.336917 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336899    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:02.336917 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336916    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:02.336979 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336921    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:02.336979 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336924    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:02.336979 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336928    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:02.336979 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336931    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:02.336979 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336934    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:02.336979 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336938    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:02.336979 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336941    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:02.336979 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336944    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:02.336979 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336946    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:02.336979 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336949    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:02.336979 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336952    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:02.336979 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336954    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:02.336979 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336957    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:02.336979 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336959    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:02.336979 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336963    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:02.336979 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336966    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:02.336979 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336968    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:02.336979 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336971    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:02.336979 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336973    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336983    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336987    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336990    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336993    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336995    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.336998    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337001    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337003    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337006    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337008    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337010    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337013    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337015    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337018    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337020    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337023    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337025    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337028    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337031    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:02.337414 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337033    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337035    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337038    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337040    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337042    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337046    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337048    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337051    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337054    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337057    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337059    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337061    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337064    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337066    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337070    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337073    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337076    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337078    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337081    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337083    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:02.337941 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337086    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:02.338425 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337090    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:02.338425 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337094    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:02.338425 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337097    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:02.338425 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337100    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:02.338425 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337103    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:02.338425 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337105    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:02.338425 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337108    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:02.338425 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337110    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:02.338425 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337113    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:02.338425 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337115    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:02.338425 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337118    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:02.338425 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337120    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:02.338425 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337123    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:02.338425 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337126    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:02.338425 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337128    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:02.338425 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337132    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:02.338425 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337135    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:02.338425 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337138    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337140    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337144    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337147    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337150    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337152    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337155    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337157    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337161    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337164    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337167    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337169    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337172    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337174    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337178    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337181    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337184    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337187    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337189    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337192    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:02.338874 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337194    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:02.339392 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337918    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:02.339392 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337926    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:02.339392 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337929    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:02.339392 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337932    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:02.339392 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337935    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:02.339392 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337938    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:02.339392 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337940    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:02.339392 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337943    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:02.339392 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337946    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:02.339392 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337949    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:02.339392 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337952    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:02.339392 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337955    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:02.339392 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337958    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:02.339392 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337960    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:02.339392 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337965    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:02.339392 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337969    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:02.339392 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337972    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:02.339392 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337976    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:02.339392 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337979    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337983    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337986    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337989    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337992    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337995    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.337998    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338000    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338003    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338006    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338009    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338011    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338014    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338017    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338019    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338022    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338025    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338027    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338030    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338032    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:02.339865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338035    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:02.340348 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338038    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:02.340348 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338040    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:02.340348 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338043    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:02.340348 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338045    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:02.340348 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338049    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:02.340348 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338051    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:02.340348 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338054    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:02.340348 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338056    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:02.340348 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338059    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:02.340348 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338062    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:02.340348 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338065    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:02.340348 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338067    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:02.340348 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338071    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:02.340348 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338074    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:02.340348 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338077    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:02.340348 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338080    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:02.340348 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338084    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:02.340348 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338088    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:02.340348 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338092    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338095    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338098    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338101    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338106    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338108    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338111    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338114    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338116    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338119    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338122    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338124    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338127    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338130    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338133    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338135    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338138    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338141    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338144    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338147    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:02.340819 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338149    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338152    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338155    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338159    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338162    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338164    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338167    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338170    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338172    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338175    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338178    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338181    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338184    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338186    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338189    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338192    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338194    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338197    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338199    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338202    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:02.341349 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338205    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:02.341865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338207    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:02.341865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338579    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:02.341865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338583    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:02.341865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338586    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:02.341865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338589    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:02.341865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338592    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:02.341865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338595    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:02.341865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338598    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:02.341865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338601    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:02.341865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338603    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:02.341865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338606    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:02.341865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338609    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:02.341865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338611    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:02.341865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338614    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:02.341865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338617    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:02.341865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338620    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:02.341865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338623    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:02.341865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338627    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:02.341865 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338631    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338634    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338638    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338641    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338644    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338647    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338650    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338653    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338656    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338658    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338661    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338664    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338666    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338669    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338672    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338675    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338677    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338680    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338683    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338686    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:02.342336 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338689    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:02.342854 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338693    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:02.342854 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338697    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:02.342854 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338700    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:02.342854 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338703    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:02.342854 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338705    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:02.342854 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338708    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:02.342854 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338711    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:02.342854 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338714    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:02.342854 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338717    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:02.342854 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338720    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:02.342854 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338724    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:02.342854 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338728    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:02.342854 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338730    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:02.342854 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338733    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:02.342854 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338736    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:02.342854 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338740    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:02.342854 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338743    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:02.342854 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338746    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:02.342854 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338748    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338751    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338754    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338757    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338760    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338763    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338765    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338768    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338771    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338773    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338776    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338779    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338781    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338786    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338789    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338792    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338795    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338798    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338800    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338803    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:02.343327 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338806    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338809    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338811    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338814    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338817    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338821    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338823    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338826    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338829    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338832    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338838    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338841    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338844    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338847    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338849    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338852    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338855    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338857    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338860    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338863    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:02.343838 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338865    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338868    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.338871    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.338951    2576 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.338985    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.338992    2576 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.338997    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339001    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339004    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339008    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339013    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339018    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339021    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339024    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339028    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339031    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339034    2576 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339037    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339040    2576 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339043    2576 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339046    2576 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339050    2576 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339053    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339057    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:41:02.344314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339060    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339063    2576 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339067    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339070    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339074    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339077    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339080    2576 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339083    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339085    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339088    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339091    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339095    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339098    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339101    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339105    2576 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339108    2576 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339111    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339115    2576 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339118    2576 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339123    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339126    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339129    2576 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339133    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339136    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339139    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:41:02.344891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339142    2576 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339145    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339147    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339150    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339153    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339156    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339159    2576 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339162    2576 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339166    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339169    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339173    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339176    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339179    2576 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339182    2576 flags.go:64] FLAG: --help="false"
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339185    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-136-127.ec2.internal"
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339188    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339190    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339193    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339196    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339200    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339203    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339206    2576 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339209    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:41:02.345481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339211    2576 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339214    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339218    2576 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339224    2576 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339227    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339230    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339234    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339236    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339239    2576 flags.go:64] FLAG: --lock-file=""
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339242    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339245    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339248    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339254    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339257    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339260    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339263    2576 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339266    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339270    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339273    2576 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339275    2576 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339280    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339283    2576 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339287    2576 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339289    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339293    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:41:02.346130 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339296    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339299    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339302    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339305    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339308    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339315    2576 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339318    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339321    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339324    2576 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339327    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339333    2576 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339337    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339340    2576 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339343    2576 flags.go:64] FLAG: --port="10250"
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339346    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339349    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09c5f65bbe4d54c75"
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339353    2576 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339356    2576 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339358    2576 flags.go:64] FLAG: --register-node="true"
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339361    2576 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339365    2576 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339368    2576 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339371    2576 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339373    2576 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339377    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:41:02.346763 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339381    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339384    2576 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339387    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339390    2576 flags.go:64] FLAG: --runonce="false"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339393    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339396    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339399    2576 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339402    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339405    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339408    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339411    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339416    2576 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339418    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339422    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339424    2576 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339427    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339430    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339433    2576 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339437    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339443    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339446    2576 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339449    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339452    2576 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339455    2576 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339458    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:41:02.347406 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339461    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339464    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339468    2576 flags.go:64] FLAG: --v="2"
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339474    2576 flags.go:64] FLAG: --version="false"
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339480    2576 flags.go:64] FLAG: --vmodule=""
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339485    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339490    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339616    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339621    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339624    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339627    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339630    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339632    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339635    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339638    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339642    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339645    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339650    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339654    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339657    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339659    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:02.348209 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339662    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339665    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339667    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339671    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339675    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339678    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339681    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339683    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339686    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339689    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339691    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339694    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339696    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339699    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339701    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339704    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339707    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339709    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339712    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339715    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:02.348944 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339718    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339721    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339724    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339726    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339729    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339731    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339734    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339736    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339740    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339746    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339749    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339752    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339755    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339758    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339760    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339763    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339766    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339769    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339772    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339774    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:02.349627 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339777    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339780    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339782    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339785    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339787    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339790    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339793    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339795    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339798    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339800    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339803    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339806    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339808    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339811    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339814    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339817    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339819    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339822    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339824    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339827    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:02.350158 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339829    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:02.350673 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339834    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:02.350673 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339838    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:02.350673 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339842    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:02.350673 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339845    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:02.350673 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339847    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:02.350673 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339850    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:02.350673 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339853    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:02.350673 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339858    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:02.350673 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339861    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:02.350673 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339864    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:02.350673 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339867    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:02.350673 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339869    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:02.350673 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339872    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:02.350673 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339875    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:02.350673 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339878    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:02.350673 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339881    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:02.350673 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339883    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:02.350673 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339886    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:02.350673 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339888    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:02.351152 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339891    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:02.351152 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339893    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:02.351152 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339896    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:02.351152 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339899    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:02.351152 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.339901    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:02.351152 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.339910    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:41:02.351152 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.349411    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:41:02.351152 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.349426    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:41:02.351152 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.349448    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:41:02.351152 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.349458    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:41:02.351152 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349516    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:02.351152 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349521    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:02.351152 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349525    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:02.351152 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349529    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:02.351152 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349532    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349537    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349542    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349546    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349550    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349555    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349576    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349580    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349583    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349586    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349589    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349592    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349594    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349597    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349600    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349603    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349607    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349609    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349612    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349615    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:02.351511 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349617    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349620    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349622    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349625    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349628    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349631    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349634    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349637    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349639    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349642    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349645    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349648    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349650    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349653    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349655    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349658    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349661    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349664    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349666    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349669    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:02.351999 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349672    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:02.352492 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349675    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:02.352492 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349679    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:02.352492 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349682    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:02.352492 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349685    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:02.352492 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349696    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:02.352492 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349699    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:02.352492 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349702    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:02.352492 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349705    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:02.352492 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349707    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:02.352492 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349710    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:02.352492 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349713    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:02.352492 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349715    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:02.352492 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349718    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:02.352492 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349721    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:02.352492 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349724    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:02.352492 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349745    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:02.352492 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349749    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:02.352492 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349751    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:02.352492 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349754    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:02.352954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349758    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:02.352954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349760    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:02.352954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349763    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:02.352954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349766    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:02.352954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349770    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:02.352954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349774    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:02.352954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349777    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:02.352954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349781    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:02.352954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349784    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:02.352954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349787    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:02.352954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349790    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:02.352954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349793    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:02.352954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349796    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:02.352954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349798    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:02.352954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349801    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:02.352954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349805    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:02.352954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349808    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:02.352954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349810    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:02.352954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349813    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:02.353460 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349815    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:02.353460 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349818    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:02.353460 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349822    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:02.353460 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349825    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:02.353460 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349828    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:02.353460 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349830    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:02.353460 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349833    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:02.353460 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349836    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:02.353460 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349838    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:02.353460 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349841    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:02.353460 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349843    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:02.353460 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349846    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:02.353460 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349850    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:02.353460 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349854    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:02.353460 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349857    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:02.353460 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349861    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:02.353902 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.349866    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:41:02.353902 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.349998    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:41:02.353902 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350004    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:41:02.353902 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350008    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:41:02.353902 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350011    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:41:02.353902 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350014    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:41:02.353902 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350017    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:41:02.353902 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350020    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:41:02.353902 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350023    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:41:02.353902 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350026    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:41:02.353902 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350029    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:41:02.353902 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350032    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:41:02.353902 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350035    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:41:02.353902 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350039    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:41:02.353902 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350042    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350045    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350048    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350051    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350053    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350056    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350059    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350061    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350064    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350067    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350070    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350072    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350075    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350077    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350080    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350083    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350085    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350088    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350090    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350093    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:41:02.354520 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350096    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:41:02.355318 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350098    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:41:02.355318 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350101    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:41:02.355318 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350104    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:41:02.355318 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350106    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:41:02.355318 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350109    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:41:02.355318 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350113    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:41:02.355318 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350117    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:41:02.355318 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350121    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:41:02.355318 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350123    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:41:02.355318 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350127    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:41:02.355318 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350130    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:41:02.355318 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350133    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:41:02.355318 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350135    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:41:02.355318 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350138    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:41:02.355318 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350140    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:41:02.355318 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350143    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:41:02.355318 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350146    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:41:02.355318 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350149    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:41:02.355318 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350151    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:41:02.355846 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350154    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:41:02.355846 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350157    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:41:02.355846 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350159    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:41:02.355846 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350162    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:41:02.355846 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350165    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:41:02.355846 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350168    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:41:02.355846 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350171    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:41:02.355846 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350175    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:41:02.355846 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350177    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:41:02.355846 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350180    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:41:02.355846 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350183    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:41:02.355846 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350186    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:41:02.355846 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350188    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:41:02.355846 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350191    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:41:02.355846 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350194    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:41:02.355846 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350196    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:41:02.355846 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350199    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:41:02.355846 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350203    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:41:02.355846 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350206    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:41:02.356363 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350209    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:41:02.356363 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350213    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:41:02.356363 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350216    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:41:02.356363 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350219    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:41:02.356363 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350222    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:41:02.356363 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350224    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:41:02.356363 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350228    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:41:02.356363 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350230    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:41:02.356363 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350233    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:41:02.356363 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350236    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:41:02.356363 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350238    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:41:02.356363 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350241    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:41:02.356363 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350246    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:41:02.356363 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350249    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:41:02.356363 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350252    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:41:02.356363 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350254    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:41:02.356363 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350257    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:41:02.356363 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350260    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:41:02.356363 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350263    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:41:02.356889 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350265    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:41:02.356889 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350268    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:41:02.356889 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350270    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:41:02.356889 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350273    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:41:02.356889 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350276    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:41:02.356889 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350279    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:41:02.356889 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350281    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:41:02.356889 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:02.350284    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:41:02.356889 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.350291    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:41:02.356889 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.350398    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:41:02.356889 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.352995    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:41:02.356889 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.353826    2576 server.go:1019] "Starting client certificate rotation"
Sep 17 17:41:02.356889 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.353934    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:41:02.356889 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.353986    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:41:02.357240 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.356099    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:41:02.357240 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.357232    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:41:02.361913 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.361887    2576 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:41:02.367696 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.367680    2576 log.go:25] "Validated CRI v1 image API"
Sep 17 17:41:02.368521 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.368509    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:41:02.371960 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.371938    2576 fs.go:135] Filesystem UUIDs: map[6ff50359-da00-4ca5-af98-562efaa76374:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c28b2403-f8e2-4f60-8bd2-8e6aa7b42b95:/dev/nvme0n1p4]
Sep 17 17:41:02.372014 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.371960    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:41:02.376882 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.376781    2576 manager.go:216] Machine: {Timestamp:2026-09-17 17:41:02.376277632 +0000 UTC m=+1.969114604 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3190551 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2750155b19636e74756480cef34bae SystemUUID:ec275015-5b19-636e-7475-6480cef34bae BootID:81b1109c-47be-401c-9737-0c754a238407 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f2:77:93:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f2:77:93:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:e7:4d:86:5a:53 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:41:02.376882 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.376879    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:41:02.377015 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.376979    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:41:02.378486 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.378462    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:41:02.378668 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.378488    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-127.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:41:02.379716 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.379695    2576 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:41:02.379716 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.379718    2576 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:41:02.379790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.379727    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:41:02.379790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.379736    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:41:02.380818 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.380808    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:41:02.383160 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.383147    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:41:02.383280 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.383271    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:41:02.383315 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.383306    2576 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:41:02.383349 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.383321    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:41:02.383349 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.383332    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:41:02.383349 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.383341    2576 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:41:02.383349 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.383350    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:41:02.383757 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.383737    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:41:02.383757 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.383758    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:41:02.387223 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.386615    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:41:02.390472 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.390451    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:41:02.390551 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.390504    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:41:02.390726 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.390715    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:41:02.390759 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.390731    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:41:02.390759 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.390737    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:41:02.390759 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.390742    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:41:02.390759 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.390750    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:41:02.390759 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.390756    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:41:02.391345 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.391337    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:41:02.391378 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.391345    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:41:02.391378 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.391352    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:41:02.391378 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.391358    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:41:02.391378 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.391366    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:41:02.391378 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.391374    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:41:02.391499 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.391402    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:41:02.391499 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.391408    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:41:02.395292 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.395280    2576 server.go:1268] "Started kubelet"
Sep 17 17:41:02.395387 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.395364    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:41:02.395445 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.395355    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:41:02.395445 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.395424    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:41:02.395752 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.395735    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:41:02.396180 ip-10-0-136-127 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:41:02.396726 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.396653    2576 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:41:02.401326 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.401302    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:41:02.403839 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.403820    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:41:02.403839 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.403831    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:41:02.403992 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.403915    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-127.ec2.internal\" not found"
Sep 17 17:41:02.404633 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.404616    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:41:02.404770 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.404636    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:41:02.404770 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.404614    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:41:02.404770 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.404730    2576 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:41:02.404770 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.404736    2576 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:41:02.404946 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.404823    2576 factory.go:55] Registering systemd factory
Sep 17 17:41:02.404946 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.404837    2576 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:41:02.405054 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.405031    2576 factory.go:153] Registering CRI-O factory
Sep 17 17:41:02.405098 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.405057    2576 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:41:02.405144 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.405121    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:41:02.405144 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.405138    2576 factory.go:103] Registering Raw factory
Sep 17 17:41:02.405237 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.405148    2576 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:41:02.405553 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.405541    2576 manager.go:349] Starting recovery of all containers
Sep 17 17:41:02.408801 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.408775    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-127.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:41:02.409523 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.409499    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:41:02.409616 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.409595    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-127.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:41:02.409697 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.409649    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-127.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:41:02.410102 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.410080    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:41:02.430364 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.430341    2576 manager.go:354] Recovery completed
Sep 17 17:41:02.432965 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.432944    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:41:02.434264 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.433475    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-127.ec2.internal.18d62c877f5b5eab  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-127.ec2.internal,UID:ip-10-0-136-127.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-127.ec2.internal,},FirstTimestamp:2026-09-17 17:41:02.395252395 +0000 UTC m=+1.988089366,LastTimestamp:2026-09-17 17:41:02.395252395 +0000 UTC m=+1.988089366,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-127.ec2.internal,}"
Sep 17 17:41:02.435742 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.435726    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:02.437856 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.437839    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:02.437940 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.437878    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:02.437940 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.437890    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:02.438437 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.438419    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:41:02.438437 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.438435    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:41:02.438525 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.438453    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:41:02.438525 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.438454    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:02.439421 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.439402    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:02.439477 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.439426    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:02.439477 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.439436    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:02.439691 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.439679    2576 policy_none.go:49] "None policy: Start"
Sep 17 17:41:02.439744 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.439696    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:41:02.439744 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.439710    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:41:02.440876 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.440856    2576 policy_none.go:47] "Start"
Sep 17 17:41:02.451004 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.450922    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-127.ec2.internal.18d62c8781e57117  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-127.ec2.internal,UID:ip-10-0-136-127.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-127.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-127.ec2.internal,},FirstTimestamp:2026-09-17 17:41:02.437855511 +0000 UTC m=+2.030692483,LastTimestamp:2026-09-17 17:41:02.437855511 +0000 UTC m=+2.030692483,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-127.ec2.internal,}"
Sep 17 17:41:02.478062 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.464713    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-127.ec2.internal.18d62c8781e5e00a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-127.ec2.internal,UID:ip-10-0-136-127.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-127.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-127.ec2.internal,},FirstTimestamp:2026-09-17 17:41:02.437883914 +0000 UTC m=+2.030720887,LastTimestamp:2026-09-17 17:41:02.437883914 +0000 UTC m=+2.030720887,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-127.ec2.internal,}"
Sep 17 17:41:02.478062 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.472913    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lq8dj"
Sep 17 17:41:02.481949 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.481885    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-127.ec2.internal.18d62c8781e6090d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-127.ec2.internal,UID:ip-10-0-136-127.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-127.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-127.ec2.internal,},FirstTimestamp:2026-09-17 17:41:02.437894413 +0000 UTC m=+2.030731387,LastTimestamp:2026-09-17 17:41:02.437894413 +0000 UTC m=+2.030731387,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-127.ec2.internal,}"
Sep 17 17:41:02.485188 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.485172    2576 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:41:02.485273 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.485210    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:41:02.485273 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.485223    2576 server.go:87] "Starting device plugin registration server"
Sep 17 17:41:02.485530 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.485517    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:41:02.485588 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.485547    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:41:02.485706 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.485682    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:41:02.485814 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.485797    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:41:02.485814 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.485813    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:41:02.486392 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.486369    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:41:02.486487 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.486448    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-127.ec2.internal\" not found"
Sep 17 17:41:02.487072 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.487057    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lq8dj"
Sep 17 17:41:02.493152 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.493087    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-127.ec2.internal.18d62c8781e57117\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-127.ec2.internal.18d62c8781e57117  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-127.ec2.internal,UID:ip-10-0-136-127.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-127.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-127.ec2.internal,},FirstTimestamp:2026-09-17 17:41:02.437855511 +0000 UTC m=+2.030692483,LastTimestamp:2026-09-17 17:41:02.439414517 +0000 UTC m=+2.032251488,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-127.ec2.internal,}"
Sep 17 17:41:02.530256 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.530224    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:41:02.531358 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.531336    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:41:02.531425 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.531400    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:41:02.531469 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.531426    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:41:02.531513 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.531472    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:41:02.534152 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.534130    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:02.586450 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.586382    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:02.587415 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.587401    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:02.587488 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.587430    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:02.587488 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.587442    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:02.587488 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.587465    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-127.ec2.internal"
Sep 17 17:41:02.597827 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.597810    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-127.ec2.internal"
Sep 17 17:41:02.597877 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.597830    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-127.ec2.internal\": node \"ip-10-0-136-127.ec2.internal\" not found"
Sep 17 17:41:02.626774 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.626750    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-127.ec2.internal\" not found"
Sep 17 17:41:02.631659 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.631639    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-127.ec2.internal"]
Sep 17 17:41:02.631722 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.631703    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:02.632762 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.632746    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:02.632843 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.632771    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:02.632843 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.632781    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:02.634208 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.634195    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:02.634343 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.634323    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal"
Sep 17 17:41:02.634379 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.634350    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:02.635665 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.635641    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:02.635665 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.635657    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:02.635791 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.635675    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:02.635791 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.635689    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:02.635791 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.635676    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:02.635791 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.635752    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:02.636884 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.636871    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-127.ec2.internal"
Sep 17 17:41:02.636925 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.636894    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:02.637707 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.637686    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:02.637813 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.637716    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:02.637813 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.637749    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:02.662142 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.662123    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-127.ec2.internal\" not found" node="ip-10-0-136-127.ec2.internal"
Sep 17 17:41:02.666597 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.666579    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-127.ec2.internal\" not found" node="ip-10-0-136-127.ec2.internal"
Sep 17 17:41:02.706882 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.706853    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e61ef62b30beabff4e4bbd49cb7c73f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal\" (UID: \"e61ef62b30beabff4e4bbd49cb7c73f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal"
Sep 17 17:41:02.706882 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.706886    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e61ef62b30beabff4e4bbd49cb7c73f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal\" (UID: \"e61ef62b30beabff4e4bbd49cb7c73f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal"
Sep 17 17:41:02.707072 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.706909    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd6c8d1b0dcecbce259dac62f1441a28-config\") pod \"kube-apiserver-proxy-ip-10-0-136-127.ec2.internal\" (UID: \"dd6c8d1b0dcecbce259dac62f1441a28\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-127.ec2.internal"
Sep 17 17:41:02.727607 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.727581    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-127.ec2.internal\" not found"
Sep 17 17:41:02.807438 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.807402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e61ef62b30beabff4e4bbd49cb7c73f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal\" (UID: \"e61ef62b30beabff4e4bbd49cb7c73f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal"
Sep 17 17:41:02.807438 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.807434    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e61ef62b30beabff4e4bbd49cb7c73f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal\" (UID: \"e61ef62b30beabff4e4bbd49cb7c73f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal"
Sep 17 17:41:02.807660 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.807460    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd6c8d1b0dcecbce259dac62f1441a28-config\") pod \"kube-apiserver-proxy-ip-10-0-136-127.ec2.internal\" (UID: \"dd6c8d1b0dcecbce259dac62f1441a28\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-127.ec2.internal"
Sep 17 17:41:02.807660 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.807491    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e61ef62b30beabff4e4bbd49cb7c73f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal\" (UID: \"e61ef62b30beabff4e4bbd49cb7c73f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal"
Sep 17 17:41:02.807660 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.807515    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e61ef62b30beabff4e4bbd49cb7c73f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal\" (UID: \"e61ef62b30beabff4e4bbd49cb7c73f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal"
Sep 17 17:41:02.807660 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.807527    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd6c8d1b0dcecbce259dac62f1441a28-config\") pod \"kube-apiserver-proxy-ip-10-0-136-127.ec2.internal\" (UID: \"dd6c8d1b0dcecbce259dac62f1441a28\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-127.ec2.internal"
Sep 17 17:41:02.828552 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.828525    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-127.ec2.internal\" not found"
Sep 17 17:41:02.929390 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:02.929308    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-127.ec2.internal\" not found"
Sep 17 17:41:02.965506 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.965475    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal"
Sep 17 17:41:02.971078 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:02.971059    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-127.ec2.internal"
Sep 17 17:41:03.029855 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:03.029817    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-127.ec2.internal\" not found"
Sep 17 17:41:03.130358 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:03.130309    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-127.ec2.internal\" not found"
Sep 17 17:41:03.231002 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:03.230905    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-127.ec2.internal\" not found"
Sep 17 17:41:03.290840 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:03.290807    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:03.331400 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:03.331355    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-127.ec2.internal\" not found"
Sep 17 17:41:03.353534 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:03.353504    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:41:03.354013 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:03.353662    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:41:03.354013 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:03.353662    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:41:03.427931 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:03.427900    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:03.431483 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:03.431456    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-127.ec2.internal\" not found"
Sep 17 17:41:03.492755 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:03.492677    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:36:02 +0000 UTC" deadline="2028-03-21 05:36:38.625077304 +0000 UTC"
Sep 17 17:41:03.492755 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:03.492718    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13211h55m35.132368532s"
Sep 17 17:41:03.532173 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:03.532134    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-127.ec2.internal\" not found"
Sep 17 17:41:03.599577 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:03.599416    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6c8d1b0dcecbce259dac62f1441a28.slice/crio-eb25bdd73b793ec070b875e47d60ee4d0d574c8e562f2c730bf2a5c8c1dee38d WatchSource:0}: Error finding container eb25bdd73b793ec070b875e47d60ee4d0d574c8e562f2c730bf2a5c8c1dee38d: Status 404 returned error can't find the container with id eb25bdd73b793ec070b875e47d60ee4d0d574c8e562f2c730bf2a5c8c1dee38d
Sep 17 17:41:03.600599 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:03.600180    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61ef62b30beabff4e4bbd49cb7c73f3.slice/crio-eb8404365011cd6088e9fc33e2d5672ecb3a4c52cb00a3a8d8be11234d99ed3c WatchSource:0}: Error finding container eb8404365011cd6088e9fc33e2d5672ecb3a4c52cb00a3a8d8be11234d99ed3c: Status 404 returned error can't find the container with id eb8404365011cd6088e9fc33e2d5672ecb3a4c52cb00a3a8d8be11234d99ed3c
Sep 17 17:41:03.607705 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:03.607684    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:41:03.632575 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:03.632521    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-127.ec2.internal\" not found"
Sep 17 17:41:03.633635 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:03.633614    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:41:03.634676 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:03.634659    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:41:03.634737 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:03.634690    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:41:03.634737 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:03.634701    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:41:03.733064 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:03.733015    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-127.ec2.internal\" not found"
Sep 17 17:41:03.792271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:03.792199    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:03.804365 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:03.804342    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal"
Sep 17 17:41:03.816228 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:03.816208    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:41:03.817179 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:03.817166    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-127.ec2.internal"
Sep 17 17:41:03.824525 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:03.824510    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:41:04.195849 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.195818    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:04.385919 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.385884    2576 apiserver.go:52] "Watching apiserver"
Sep 17 17:41:04.395051 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.395027    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:41:04.395482 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.395456    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-vmxsb","openshift-multus/multus-additional-cni-plugins-7qls8","openshift-multus/multus-rdmsd","openshift-network-diagnostics/network-check-target-qrrjr","openshift-network-operator/iptables-alerter-mh9jw","kube-system/konnectivity-agent-mb5dq","kube-system/kube-apiserver-proxy-ip-10-0-136-127.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp","openshift-image-registry/node-ca-7rmhq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal","openshift-multus/network-metrics-daemon-z44z2","openshift-ovn-kubernetes/ovnkube-node-s4nxf","kube-system/global-pull-secret-syncer-7cdxt","openshift-cluster-node-tuning-operator/tuned-j5j7r"]
Sep 17 17:41:04.397175 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.397115    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:04.397306 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:04.397230    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7cdxt" podUID="a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2"
Sep 17 17:41:04.398351 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.398328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.399331 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.399306    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.401247 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.400885    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-flss9\""
Sep 17 17:41:04.401247 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.401145    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:41:04.401392 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.401345    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:41:04.401392 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.401361    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:41:04.401777 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.401534    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:41:04.401777 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.401589    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:41:04.401777 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.401643    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:04.401777 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:04.401711    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrrjr" podUID="365faef8-3867-42a8-a227-d81293771f14"
Sep 17 17:41:04.402091 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.402074    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wfmdv\""
Sep 17 17:41:04.402239 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.402215    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:41:04.402340 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.402220    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mh9jw"
Sep 17 17:41:04.403444 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.403426    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mb5dq"
Sep 17 17:41:04.403964 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.403941    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:41:04.404625 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.404441    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5nz9z\""
Sep 17 17:41:04.404625 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.404610    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vmxsb"
Sep 17 17:41:04.404806 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.404779    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:41:04.405627 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.405606    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:41:04.405796 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.405779    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:41:04.405921 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.405831    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.406069 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.406051    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gvbv4\""
Sep 17 17:41:04.406354 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.406335    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:41:04.406951 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.406921    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:41:04.407172 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.407153    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:41:04.407397 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.407381    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dmw27\""
Sep 17 17:41:04.407736 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.407601    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:41:04.408941 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.408921    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7rmhq"
Sep 17 17:41:04.409156 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.409106    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:41:04.409378 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.409363    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:41:04.409928 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.409753    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nb8hq\""
Sep 17 17:41:04.409928 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.409778    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:41:04.410356 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.410338    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.412186 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.412144    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6l7ct\""
Sep 17 17:41:04.412281 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.412193    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:41:04.412731 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.412575    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:41:04.413528 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.412904    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:04.413528 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:04.412984    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z44z2" podUID="84810a50-2551-4d78-ac6d-4d17f1524389"
Sep 17 17:41:04.413528 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.412906    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:41:04.415034 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.415018    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.416076 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.415825    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:41:04.418271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416358    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hxzvp\""
Sep 17 17:41:04.418271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416423    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5qhs\" (UniqueName: \"kubernetes.io/projected/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-kube-api-access-r5qhs\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.418271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416453    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-host-run-netns\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.418271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-hostroot\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.418271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416499    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w7vx\" (UniqueName: \"kubernetes.io/projected/8bacc5d4-bc80-46b0-93f5-c6dd572df3c1-kube-api-access-9w7vx\") pod \"iptables-alerter-mh9jw\" (UID: \"8bacc5d4-bc80-46b0-93f5-c6dd572df3c1\") " pod="openshift-network-operator/iptables-alerter-mh9jw"
Sep 17 17:41:04.418271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416543    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0623a097-d7b1-4be4-be54-a29b81cc540a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.418271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416588    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d54d9137-d225-4657-b601-b1a8bdee3b68-hosts-file\") pod \"node-resolver-vmxsb\" (UID: \"d54d9137-d225-4657-b601-b1a8bdee3b68\") " pod="openshift-dns/node-resolver-vmxsb"
Sep 17 17:41:04.418271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416613    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/feff5f74-e1df-4fdc-b2f3-ebd27908a886-agent-certs\") pod \"konnectivity-agent-mb5dq\" (UID: \"feff5f74-e1df-4fdc-b2f3-ebd27908a886\") " pod="kube-system/konnectivity-agent-mb5dq"
Sep 17 17:41:04.418271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416634    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67858c18-1e78-4ca1-9ade-132a0e6251c7-etc-selinux\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.418271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416637    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:41:04.418271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416657    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-dbus\") pod \"global-pull-secret-syncer-7cdxt\" (UID: \"a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2\") " pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:04.418271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416721    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret\") pod \"global-pull-secret-syncer-7cdxt\" (UID: \"a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2\") " pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:04.418271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416745    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0623a097-d7b1-4be4-be54-a29b81cc540a-cni-binary-copy\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.418271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416765    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-multus-conf-dir\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.418271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416785    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-multus-cni-dir\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.418271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416809    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k26qr\" (UniqueName: \"kubernetes.io/projected/d54d9137-d225-4657-b601-b1a8bdee3b68-kube-api-access-k26qr\") pod \"node-resolver-vmxsb\" (UID: \"d54d9137-d225-4657-b601-b1a8bdee3b68\") " pod="openshift-dns/node-resolver-vmxsb"
Sep 17 17:41:04.418271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416830    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0623a097-d7b1-4be4-be54-a29b81cc540a-cnibin\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.419128 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416852    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-multus-daemon-config\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.419128 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416875    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll9v7\" (UniqueName: \"kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7\") pod \"network-check-target-qrrjr\" (UID: \"365faef8-3867-42a8-a227-d81293771f14\") " pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:04.419128 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416897    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8bacc5d4-bc80-46b0-93f5-c6dd572df3c1-iptables-alerter-script\") pod \"iptables-alerter-mh9jw\" (UID: \"8bacc5d4-bc80-46b0-93f5-c6dd572df3c1\") " pod="openshift-network-operator/iptables-alerter-mh9jw"
Sep 17 17:41:04.419128 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bacc5d4-bc80-46b0-93f5-c6dd572df3c1-host-slash\") pod \"iptables-alerter-mh9jw\" (UID: \"8bacc5d4-bc80-46b0-93f5-c6dd572df3c1\") " pod="openshift-network-operator/iptables-alerter-mh9jw"
Sep 17 17:41:04.419128 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416956    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fhv8\" (UniqueName: \"kubernetes.io/projected/67858c18-1e78-4ca1-9ade-132a0e6251c7-kube-api-access-7fhv8\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.419128 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.416973    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0623a097-d7b1-4be4-be54-a29b81cc540a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.419128 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-system-cni-dir\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.419128 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417061    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-cnibin\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.419128 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417087    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-host-run-multus-certs\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.419128 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417136    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-etc-kubernetes\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.419128 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417160    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/feff5f74-e1df-4fdc-b2f3-ebd27908a886-konnectivity-ca\") pod \"konnectivity-agent-mb5dq\" (UID: \"feff5f74-e1df-4fdc-b2f3-ebd27908a886\") " pod="kube-system/konnectivity-agent-mb5dq"
Sep 17 17:41:04.419128 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417183    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67858c18-1e78-4ca1-9ade-132a0e6251c7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.419128 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417265    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67858c18-1e78-4ca1-9ade-132a0e6251c7-socket-dir\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.419128 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417288    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-multus-socket-dir-parent\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.419128 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417314    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp9rl\" (UniqueName: \"kubernetes.io/projected/0623a097-d7b1-4be4-be54-a29b81cc540a-kube-api-access-hp9rl\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.419128 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417336    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d54d9137-d225-4657-b601-b1a8bdee3b68-tmp-dir\") pod \"node-resolver-vmxsb\" (UID: \"d54d9137-d225-4657-b601-b1a8bdee3b68\") " pod="openshift-dns/node-resolver-vmxsb"
Sep 17 17:41:04.419834 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417358    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67858c18-1e78-4ca1-9ade-132a0e6251c7-registration-dir\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.419834 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417380    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67858c18-1e78-4ca1-9ade-132a0e6251c7-device-dir\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.419834 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417402    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0623a097-d7b1-4be4-be54-a29b81cc540a-system-cni-dir\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.419834 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417449    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-cni-binary-copy\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.419834 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417484    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-host-run-k8s-cni-cncf-io\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.419834 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417506    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-host-var-lib-cni-multus\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.419834 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417533    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67858c18-1e78-4ca1-9ade-132a0e6251c7-sys-fs\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.419834 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417555    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-kubelet-config\") pod \"global-pull-secret-syncer-7cdxt\" (UID: \"a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2\") " pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:04.419834 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417597    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0623a097-d7b1-4be4-be54-a29b81cc540a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.419834 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-host-var-lib-cni-bin\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.419834 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417644    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-host-var-lib-kubelet\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.419834 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417753    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:41:04.419834 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.417889    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:41:04.419834 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.418007    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:41:04.419834 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.419794    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:41:04.419834 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.419835    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:41:04.420404 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.420022    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:41:04.420584 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.420542    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xsvs4\""
Sep 17 17:41:04.423522 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.423500    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:41:04.447152 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.447101    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wb98n"
Sep 17 17:41:04.459116 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.459093    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wb98n"
Sep 17 17:41:04.505983 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.505951    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:41:04.519111 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519083    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-hostroot\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.519271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519122    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-kubelet\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.519271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519146    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-run-ovn\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.519271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519203    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-cni-netd\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.519271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519213    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-hostroot\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.519271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519236    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-run\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.519271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d54d9137-d225-4657-b601-b1a8bdee3b68-hosts-file\") pod \"node-resolver-vmxsb\" (UID: \"d54d9137-d225-4657-b601-b1a8bdee3b68\") " pod="openshift-dns/node-resolver-vmxsb"
Sep 17 17:41:04.519523 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519330    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d54d9137-d225-4657-b601-b1a8bdee3b68-hosts-file\") pod \"node-resolver-vmxsb\" (UID: \"d54d9137-d225-4657-b601-b1a8bdee3b68\") " pod="openshift-dns/node-resolver-vmxsb"
Sep 17 17:41:04.519523 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519336    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-dbus\") pod \"global-pull-secret-syncer-7cdxt\" (UID: \"a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2\") " pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:04.519523 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519372    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0623a097-d7b1-4be4-be54-a29b81cc540a-cni-binary-copy\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.519523 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519390    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-multus-conf-dir\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.519523 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519444    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-multus-daemon-config\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.519523 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519471    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll9v7\" (UniqueName: \"kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7\") pod \"network-check-target-qrrjr\" (UID: \"365faef8-3867-42a8-a227-d81293771f14\") " pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:04.519523 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519495    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bacc5d4-bc80-46b0-93f5-c6dd572df3c1-host-slash\") pod \"iptables-alerter-mh9jw\" (UID: \"8bacc5d4-bc80-46b0-93f5-c6dd572df3c1\") " pod="openshift-network-operator/iptables-alerter-mh9jw"
Sep 17 17:41:04.519925 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519529    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-sysconfig\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.519925 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519549    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-dbus\") pod \"global-pull-secret-syncer-7cdxt\" (UID: \"a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2\") " pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:04.519925 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519628    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-multus-cni-dir\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.519925 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519620    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bacc5d4-bc80-46b0-93f5-c6dd572df3c1-host-slash\") pod \"iptables-alerter-mh9jw\" (UID: \"8bacc5d4-bc80-46b0-93f5-c6dd572df3c1\") " pod="openshift-network-operator/iptables-alerter-mh9jw"
Sep 17 17:41:04.519925 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-multus-cni-dir\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.519925 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519690    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k26qr\" (UniqueName: \"kubernetes.io/projected/d54d9137-d225-4657-b601-b1a8bdee3b68-kube-api-access-k26qr\") pod \"node-resolver-vmxsb\" (UID: \"d54d9137-d225-4657-b601-b1a8bdee3b68\") " pod="openshift-dns/node-resolver-vmxsb"
Sep 17 17:41:04.519925 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519706    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0623a097-d7b1-4be4-be54-a29b81cc540a-cnibin\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.519925 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519726    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-tuned\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.519925 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519747    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8611822c-22f3-40b6-91fd-677ce0e5ae70-tmp\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.519925 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519764    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs\") pod \"network-metrics-daemon-z44z2\" (UID: \"84810a50-2551-4d78-ac6d-4d17f1524389\") " pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:04.519925 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519788    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fhv8\" (UniqueName: \"kubernetes.io/projected/67858c18-1e78-4ca1-9ade-132a0e6251c7-kube-api-access-7fhv8\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.519925 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519815    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-system-cni-dir\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.519925 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519852    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-cnibin\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.519925 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-host-run-multus-certs\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.519925 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-etc-kubernetes\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.519925 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519918    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcbhw\" (UniqueName: \"kubernetes.io/projected/0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852-kube-api-access-hcbhw\") pod \"node-ca-7rmhq\" (UID: \"0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852\") " pod="openshift-image-registry/node-ca-7rmhq"
Sep 17 17:41:04.519925 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519932    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-sysctl-d\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.520656 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519947    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.520656 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519962    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-sys\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.520656 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.519982    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-run-systemd\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.520656 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520008    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-etc-openvswitch\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.520656 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520018    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-multus-daemon-config\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.520656 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520029    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-run-openvswitch\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.520656 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520018    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0623a097-d7b1-4be4-be54-a29b81cc540a-cni-binary-copy\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.520656 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520067    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0623a097-d7b1-4be4-be54-a29b81cc540a-cnibin\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.520656 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520069    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-log-socket\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.520656 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520105    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-env-overrides\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.520656 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520127    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-host-run-multus-certs\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.520656 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520133    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67858c18-1e78-4ca1-9ade-132a0e6251c7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.520656 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520135    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-etc-kubernetes\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.520656 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520192    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-cnibin\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.520656 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520193    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-system-cni-dir\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.520656 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520217    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67858c18-1e78-4ca1-9ade-132a0e6251c7-socket-dir\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.520656 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520243    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67858c18-1e78-4ca1-9ade-132a0e6251c7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.521432 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-multus-socket-dir-parent\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.521432 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520272    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-multus-conf-dir\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.521432 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520287    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-sysctl-conf\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.521432 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520322    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-multus-socket-dir-parent\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.521432 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520333    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67858c18-1e78-4ca1-9ade-132a0e6251c7-socket-dir\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.521432 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520364    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp9rl\" (UniqueName: \"kubernetes.io/projected/0623a097-d7b1-4be4-be54-a29b81cc540a-kube-api-access-hp9rl\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.521432 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520408    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-cni-binary-copy\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.521432 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520447    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-host-var-lib-cni-multus\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.521432 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520550    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-run-ovn-kubernetes\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.521432 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520553    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-host-var-lib-cni-multus\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.521432 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-ovnkube-script-lib\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.521432 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520679    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-host-var-lib-cni-bin\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.521432 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520704    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w7vx\" (UniqueName: \"kubernetes.io/projected/8bacc5d4-bc80-46b0-93f5-c6dd572df3c1-kube-api-access-9w7vx\") pod \"iptables-alerter-mh9jw\" (UID: \"8bacc5d4-bc80-46b0-93f5-c6dd572df3c1\") " pod="openshift-network-operator/iptables-alerter-mh9jw"
Sep 17 17:41:04.521432 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520722    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhcjt\" (UniqueName: \"kubernetes.io/projected/84810a50-2551-4d78-ac6d-4d17f1524389-kube-api-access-lhcjt\") pod \"network-metrics-daemon-z44z2\" (UID: \"84810a50-2551-4d78-ac6d-4d17f1524389\") " pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:04.521432 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520739    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-var-lib-openvswitch\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.521432 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520757    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-ovnkube-config\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.521432 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520796    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-host-run-netns\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.522170 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520813    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852-host\") pod \"node-ca-7rmhq\" (UID: \"0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852\") " pod="openshift-image-registry/node-ca-7rmhq"
Sep 17 17:41:04.522170 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-slash\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.522170 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520854    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-node-log\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.522170 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520878    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0623a097-d7b1-4be4-be54-a29b81cc540a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.522170 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520888    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-host-run-netns\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.522170 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/feff5f74-e1df-4fdc-b2f3-ebd27908a886-agent-certs\") pod \"konnectivity-agent-mb5dq\" (UID: \"feff5f74-e1df-4fdc-b2f3-ebd27908a886\") " pod="kube-system/konnectivity-agent-mb5dq"
Sep 17 17:41:04.522170 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520878    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-host-var-lib-cni-bin\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.522170 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520924    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-cni-binary-copy\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.522170 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520947    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67858c18-1e78-4ca1-9ade-132a0e6251c7-etc-selinux\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.522170 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.520973    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret\") pod \"global-pull-secret-syncer-7cdxt\" (UID: \"a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2\") " pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:04.522170 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521052    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8bacc5d4-bc80-46b0-93f5-c6dd572df3c1-iptables-alerter-script\") pod \"iptables-alerter-mh9jw\" (UID: \"8bacc5d4-bc80-46b0-93f5-c6dd572df3c1\") " pod="openshift-network-operator/iptables-alerter-mh9jw"
Sep 17 17:41:04.522170 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521057    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0623a097-d7b1-4be4-be54-a29b81cc540a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.522170 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521090    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-cni-bin\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.522170 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521100    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67858c18-1e78-4ca1-9ade-132a0e6251c7-etc-selinux\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.522170 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:04.521128    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:04.522170 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521162    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpk72\" (UniqueName: \"kubernetes.io/projected/8611822c-22f3-40b6-91fd-677ce0e5ae70-kube-api-access-rpk72\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.522170 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521191    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0623a097-d7b1-4be4-be54-a29b81cc540a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.522953 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:04.521254    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret podName:a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:05.021216285 +0000 UTC m=+4.614053248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret") pod "global-pull-secret-syncer-7cdxt" (UID: "a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:04.522953 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521277    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-var-lib-kubelet\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.522953 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521269    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:41:04.522953 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vzlh\" (UniqueName: \"kubernetes.io/projected/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-kube-api-access-5vzlh\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.522953 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521382    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852-serviceca\") pod \"node-ca-7rmhq\" (UID: \"0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852\") " pod="openshift-image-registry/node-ca-7rmhq"
Sep 17 17:41:04.522953 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521407    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-systemd\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.522953 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521430    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-lib-modules\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.522953 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521450    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-ovn-node-metrics-cert\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.522953 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521484    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/feff5f74-e1df-4fdc-b2f3-ebd27908a886-konnectivity-ca\") pod \"konnectivity-agent-mb5dq\" (UID: \"feff5f74-e1df-4fdc-b2f3-ebd27908a886\") " pod="kube-system/konnectivity-agent-mb5dq"
Sep 17 17:41:04.522953 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521508    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-kubernetes\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.522953 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521537    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-modprobe-d\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.522953 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d54d9137-d225-4657-b601-b1a8bdee3b68-tmp-dir\") pod \"node-resolver-vmxsb\" (UID: \"d54d9137-d225-4657-b601-b1a8bdee3b68\") " pod="openshift-dns/node-resolver-vmxsb"
Sep 17 17:41:04.522953 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67858c18-1e78-4ca1-9ade-132a0e6251c7-registration-dir\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.522953 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67858c18-1e78-4ca1-9ade-132a0e6251c7-device-dir\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.522953 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521671    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0623a097-d7b1-4be4-be54-a29b81cc540a-system-cni-dir\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.522953 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521700    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-host-run-k8s-cni-cncf-io\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.523503 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-host-var-lib-kubelet\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.523503 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521766    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-systemd-units\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.523503 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521770    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67858c18-1e78-4ca1-9ade-132a0e6251c7-device-dir\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.523503 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521792    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67858c18-1e78-4ca1-9ade-132a0e6251c7-sys-fs\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.523503 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-kubelet-config\") pod \"global-pull-secret-syncer-7cdxt\" (UID: \"a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2\") " pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:04.523503 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521828    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67858c18-1e78-4ca1-9ade-132a0e6251c7-registration-dir\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.523503 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521841    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0623a097-d7b1-4be4-be54-a29b81cc540a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.523503 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521877    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0623a097-d7b1-4be4-be54-a29b81cc540a-system-cni-dir\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.523503 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521915    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5qhs\" (UniqueName: \"kubernetes.io/projected/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-kube-api-access-r5qhs\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.523503 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521946    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-host\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.523503 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-run-netns\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.523503 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521979    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-host-var-lib-kubelet\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.523503 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.522092    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67858c18-1e78-4ca1-9ade-132a0e6251c7-sys-fs\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.523503 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.522133    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-kubelet-config\") pod \"global-pull-secret-syncer-7cdxt\" (UID: \"a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2\") " pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:04.523503 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.522195    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8bacc5d4-bc80-46b0-93f5-c6dd572df3c1-iptables-alerter-script\") pod \"iptables-alerter-mh9jw\" (UID: \"8bacc5d4-bc80-46b0-93f5-c6dd572df3c1\") " pod="openshift-network-operator/iptables-alerter-mh9jw"
Sep 17 17:41:04.523503 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.521846    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-host-run-k8s-cni-cncf-io\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.523503 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.522314    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0623a097-d7b1-4be4-be54-a29b81cc540a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.524097 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.522406    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d54d9137-d225-4657-b601-b1a8bdee3b68-tmp-dir\") pod \"node-resolver-vmxsb\" (UID: \"d54d9137-d225-4657-b601-b1a8bdee3b68\") " pod="openshift-dns/node-resolver-vmxsb"
Sep 17 17:41:04.524097 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.522749    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0623a097-d7b1-4be4-be54-a29b81cc540a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.524097 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.523395    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/feff5f74-e1df-4fdc-b2f3-ebd27908a886-konnectivity-ca\") pod \"konnectivity-agent-mb5dq\" (UID: \"feff5f74-e1df-4fdc-b2f3-ebd27908a886\") " pod="kube-system/konnectivity-agent-mb5dq"
Sep 17 17:41:04.525413 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.525392    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/feff5f74-e1df-4fdc-b2f3-ebd27908a886-agent-certs\") pod \"konnectivity-agent-mb5dq\" (UID: \"feff5f74-e1df-4fdc-b2f3-ebd27908a886\") " pod="kube-system/konnectivity-agent-mb5dq"
Sep 17 17:41:04.525930 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:04.525910    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:04.525930 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:04.525933    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:04.526097 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:04.525946    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ll9v7 for pod openshift-network-diagnostics/network-check-target-qrrjr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:04.526097 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:04.526006    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7 podName:365faef8-3867-42a8-a227-d81293771f14 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:05.025991868 +0000 UTC m=+4.618828853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ll9v7" (UniqueName: "kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7") pod "network-check-target-qrrjr" (UID: "365faef8-3867-42a8-a227-d81293771f14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:04.529576 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.529547    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fhv8\" (UniqueName: \"kubernetes.io/projected/67858c18-1e78-4ca1-9ade-132a0e6251c7-kube-api-access-7fhv8\") pod \"aws-ebs-csi-driver-node-npsqp\" (UID: \"67858c18-1e78-4ca1-9ade-132a0e6251c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.533787 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.533721    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp9rl\" (UniqueName: \"kubernetes.io/projected/0623a097-d7b1-4be4-be54-a29b81cc540a-kube-api-access-hp9rl\") pod \"multus-additional-cni-plugins-7qls8\" (UID: \"0623a097-d7b1-4be4-be54-a29b81cc540a\") " pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.533967 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.533942    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k26qr\" (UniqueName: \"kubernetes.io/projected/d54d9137-d225-4657-b601-b1a8bdee3b68-kube-api-access-k26qr\") pod \"node-resolver-vmxsb\" (UID: \"d54d9137-d225-4657-b601-b1a8bdee3b68\") " pod="openshift-dns/node-resolver-vmxsb"
Sep 17 17:41:04.534072 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.533942    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5qhs\" (UniqueName: \"kubernetes.io/projected/28657902-dde3-4d9b-bbaa-ce6abfefc4e9-kube-api-access-r5qhs\") pod \"multus-rdmsd\" (UID: \"28657902-dde3-4d9b-bbaa-ce6abfefc4e9\") " pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.535630 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.535606    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w7vx\" (UniqueName: \"kubernetes.io/projected/8bacc5d4-bc80-46b0-93f5-c6dd572df3c1-kube-api-access-9w7vx\") pod \"iptables-alerter-mh9jw\" (UID: \"8bacc5d4-bc80-46b0-93f5-c6dd572df3c1\") " pod="openshift-network-operator/iptables-alerter-mh9jw"
Sep 17 17:41:04.536775 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.536727    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-127.ec2.internal" event={"ID":"dd6c8d1b0dcecbce259dac62f1441a28","Type":"ContainerStarted","Data":"eb25bdd73b793ec070b875e47d60ee4d0d574c8e562f2c730bf2a5c8c1dee38d"}
Sep 17 17:41:04.540696 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.538537    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal" event={"ID":"e61ef62b30beabff4e4bbd49cb7c73f3","Type":"ContainerStarted","Data":"eb8404365011cd6088e9fc33e2d5672ecb3a4c52cb00a3a8d8be11234d99ed3c"}
Sep 17 17:41:04.623081 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623041    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-cni-bin\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.623081 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623082    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpk72\" (UniqueName: \"kubernetes.io/projected/8611822c-22f3-40b6-91fd-677ce0e5ae70-kube-api-access-rpk72\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.623327 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623098    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-var-lib-kubelet\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.623327 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623118    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzlh\" (UniqueName: \"kubernetes.io/projected/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-kube-api-access-5vzlh\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.623327 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623141    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852-serviceca\") pod \"node-ca-7rmhq\" (UID: \"0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852\") " pod="openshift-image-registry/node-ca-7rmhq"
Sep 17 17:41:04.623327 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-systemd\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.623327 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623170    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-cni-bin\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.623327 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623200    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-var-lib-kubelet\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.623327 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623182    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-lib-modules\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.623327 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-ovn-node-metrics-cert\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.623327 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623276    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-kubernetes\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.623327 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623280    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-systemd\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.623327 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623300    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-modprobe-d\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.623327 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623316    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-lib-modules\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.623327 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623331    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-systemd-units\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.623944 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623362    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-host\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.623944 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623366    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-kubernetes\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.623944 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623387    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-run-netns\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.623944 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623404    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-systemd-units\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.623944 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623411    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-kubelet\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.623944 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623436    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-run-ovn\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.623944 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623438    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-modprobe-d\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.623944 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623452    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-host\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.623944 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623458    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-cni-netd\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.623944 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623488    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-run-netns\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.623944 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623493    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-kubelet\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.623944 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623495    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-run-ovn\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.623944 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623506    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-cni-netd\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.623944 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623529    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-run\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.623944 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623596    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-sysconfig\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.623944 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623624    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-tuned\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.623944 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623666    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8611822c-22f3-40b6-91fd-677ce0e5ae70-tmp\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.623944 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623680    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-sysconfig\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.624790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623635    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-run\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.624790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623686    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs\") pod \"network-metrics-daemon-z44z2\" (UID: \"84810a50-2551-4d78-ac6d-4d17f1524389\") " pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:04.624790 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:04.623759    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:04.624790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623762    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcbhw\" (UniqueName: \"kubernetes.io/projected/0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852-kube-api-access-hcbhw\") pod \"node-ca-7rmhq\" (UID: \"0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852\") " pod="openshift-image-registry/node-ca-7rmhq"
Sep 17 17:41:04.624790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623714    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852-serviceca\") pod \"node-ca-7rmhq\" (UID: \"0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852\") " pod="openshift-image-registry/node-ca-7rmhq"
Sep 17 17:41:04.624790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-sysctl-d\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.624790 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:04.623828    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs podName:84810a50-2551-4d78-ac6d-4d17f1524389 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:05.123809734 +0000 UTC m=+4.716646715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs") pod "network-metrics-daemon-z44z2" (UID: "84810a50-2551-4d78-ac6d-4d17f1524389") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:04.624790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.624790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.623992    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-sys\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.624790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624015    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-run-systemd\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.624790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624044    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-etc-openvswitch\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.624790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624071    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-run-openvswitch\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.624790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624075    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.624790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624089    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-sys\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.624790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624008    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-sysctl-d\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.624790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-log-socket\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.624790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624136    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-log-socket\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.625605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624136    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-etc-openvswitch\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.625605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624161    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-run-systemd\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.625605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624173    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-run-openvswitch\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.625605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624160    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-env-overrides\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.625605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-sysctl-conf\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.625605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-run-ovn-kubernetes\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.625605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-ovnkube-script-lib\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.625605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624374    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhcjt\" (UniqueName: \"kubernetes.io/projected/84810a50-2551-4d78-ac6d-4d17f1524389-kube-api-access-lhcjt\") pod \"network-metrics-daemon-z44z2\" (UID: \"84810a50-2551-4d78-ac6d-4d17f1524389\") " pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:04.625605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624394    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-var-lib-openvswitch\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.625605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624396    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-run-ovn-kubernetes\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.625605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624421    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-ovnkube-config\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.625605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852-host\") pod \"node-ca-7rmhq\" (UID: \"0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852\") " pod="openshift-image-registry/node-ca-7rmhq"
Sep 17 17:41:04.625605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-slash\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.625605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624479    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-sysctl-conf\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.625605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624485    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-node-log\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.625605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624516    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-node-log\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.625605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624551    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-var-lib-openvswitch\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.626315 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624606    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-env-overrides\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.626315 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624613    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852-host\") pod \"node-ca-7rmhq\" (UID: \"0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852\") " pod="openshift-image-registry/node-ca-7rmhq"
Sep 17 17:41:04.626315 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.624644    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-host-slash\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.626315 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.625022    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-ovnkube-config\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.626315 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.625419    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-ovnkube-script-lib\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.626315 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.625978    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-ovn-node-metrics-cert\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.626315 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.626037    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8611822c-22f3-40b6-91fd-677ce0e5ae70-tmp\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.626315 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.626164    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8611822c-22f3-40b6-91fd-677ce0e5ae70-etc-tuned\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.643016 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.642983    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpk72\" (UniqueName: \"kubernetes.io/projected/8611822c-22f3-40b6-91fd-677ce0e5ae70-kube-api-access-rpk72\") pod \"tuned-j5j7r\" (UID: \"8611822c-22f3-40b6-91fd-677ce0e5ae70\") " pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.643233 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.643213    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vzlh\" (UniqueName: \"kubernetes.io/projected/5bc15f81-98ec-416f-a1bd-9f5d6d2a367b-kube-api-access-5vzlh\") pod \"ovnkube-node-s4nxf\" (UID: \"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:04.643920 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.643895    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcbhw\" (UniqueName: \"kubernetes.io/projected/0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852-kube-api-access-hcbhw\") pod \"node-ca-7rmhq\" (UID: \"0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852\") " pod="openshift-image-registry/node-ca-7rmhq"
Sep 17 17:41:04.644043 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.643923    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhcjt\" (UniqueName: \"kubernetes.io/projected/84810a50-2551-4d78-ac6d-4d17f1524389-kube-api-access-lhcjt\") pod \"network-metrics-daemon-z44z2\" (UID: \"84810a50-2551-4d78-ac6d-4d17f1524389\") " pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:04.713779 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.713668    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7qls8"
Sep 17 17:41:04.722586 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.722543    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rdmsd"
Sep 17 17:41:04.734451 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.734424    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mh9jw"
Sep 17 17:41:04.738739 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.738707    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mb5dq"
Sep 17 17:41:04.745313 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.745288    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vmxsb"
Sep 17 17:41:04.751152 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.751079    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp"
Sep 17 17:41:04.757752 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.757709    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7rmhq"
Sep 17 17:41:04.763729 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.763710    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j5j7r"
Sep 17 17:41:04.768931 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:04.768905    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:05.027743 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:05.027661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll9v7\" (UniqueName: \"kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7\") pod \"network-check-target-qrrjr\" (UID: \"365faef8-3867-42a8-a227-d81293771f14\") " pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:05.027743 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:05.027730    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret\") pod \"global-pull-secret-syncer-7cdxt\" (UID: \"a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2\") " pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:05.027914 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:05.027829    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:05.027914 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:05.027847    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:05.027914 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:05.027855    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ll9v7 for pod openshift-network-diagnostics/network-check-target-qrrjr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:05.027914 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:05.027869    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:05.027914 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:05.027911    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7 podName:365faef8-3867-42a8-a227-d81293771f14 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:06.027894572 +0000 UTC m=+5.620731554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll9v7" (UniqueName: "kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7") pod "network-check-target-qrrjr" (UID: "365faef8-3867-42a8-a227-d81293771f14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:05.028056 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:05.027932    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret podName:a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:06.027920347 +0000 UTC m=+5.620757310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret") pod "global-pull-secret-syncer-7cdxt" (UID: "a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:05.128734 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:05.128692    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs\") pod \"network-metrics-daemon-z44z2\" (UID: \"84810a50-2551-4d78-ac6d-4d17f1524389\") " pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:05.128921 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:05.128859    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:05.128982 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:05.128942    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs podName:84810a50-2551-4d78-ac6d-4d17f1524389 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:06.128920851 +0000 UTC m=+5.721757814 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs") pod "network-metrics-daemon-z44z2" (UID: "84810a50-2551-4d78-ac6d-4d17f1524389") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:05.346082 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:05.346043    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67858c18_1e78_4ca1_9ade_132a0e6251c7.slice/crio-e38a6bf605bb4ca17cd92d0529acc5bc93258a3f006211a1814bfda09ccd1419 WatchSource:0}: Error finding container e38a6bf605bb4ca17cd92d0529acc5bc93258a3f006211a1814bfda09ccd1419: Status 404 returned error can't find the container with id e38a6bf605bb4ca17cd92d0529acc5bc93258a3f006211a1814bfda09ccd1419
Sep 17 17:41:05.348611 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:05.348579    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc15f81_98ec_416f_a1bd_9f5d6d2a367b.slice/crio-7fceb6d232c3e319b074e14118cd72bdc147375cd630d5a3e11d652045a50d27 WatchSource:0}: Error finding container 7fceb6d232c3e319b074e14118cd72bdc147375cd630d5a3e11d652045a50d27: Status 404 returned error can't find the container with id 7fceb6d232c3e319b074e14118cd72bdc147375cd630d5a3e11d652045a50d27
Sep 17 17:41:05.350419 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:05.350188    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bacc5d4_bc80_46b0_93f5_c6dd572df3c1.slice/crio-b319992c0bca6b9fd4f05c512833e2fcb16dc0162f6c81d00eee61ba29e0406c WatchSource:0}: Error finding container b319992c0bca6b9fd4f05c512833e2fcb16dc0162f6c81d00eee61ba29e0406c: Status 404 returned error can't find the container with id b319992c0bca6b9fd4f05c512833e2fcb16dc0162f6c81d00eee61ba29e0406c
Sep 17 17:41:05.353157 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:05.353137    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0fc3cd_4a4b_4fd3_ad2b_aa9865f74852.slice/crio-e36373421344352256402610de03757c27307c533ab94c493954a2701dcbde3e WatchSource:0}: Error finding container e36373421344352256402610de03757c27307c533ab94c493954a2701dcbde3e: Status 404 returned error can't find the container with id e36373421344352256402610de03757c27307c533ab94c493954a2701dcbde3e
Sep 17 17:41:05.353907 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:05.353886    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28657902_dde3_4d9b_bbaa_ce6abfefc4e9.slice/crio-8abc6b82a4e732604818b031f9dd098aab87b8cbd6ab5b64948e342f6990abda WatchSource:0}: Error finding container 8abc6b82a4e732604818b031f9dd098aab87b8cbd6ab5b64948e342f6990abda: Status 404 returned error can't find the container with id 8abc6b82a4e732604818b031f9dd098aab87b8cbd6ab5b64948e342f6990abda
Sep 17 17:41:05.354867 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:05.354767    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeff5f74_e1df_4fdc_b2f3_ebd27908a886.slice/crio-78107dbba03764b1772d374421f026517605918a6f4d6b447106e90bac980911 WatchSource:0}: Error finding container 78107dbba03764b1772d374421f026517605918a6f4d6b447106e90bac980911: Status 404 returned error can't find the container with id 78107dbba03764b1772d374421f026517605918a6f4d6b447106e90bac980911
Sep 17 17:41:05.355954 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:05.355930    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0623a097_d7b1_4be4_be54_a29b81cc540a.slice/crio-451d1c05cb8c8a01ad706231fb884a50320bbb0adaea5d857a109badd8364271 WatchSource:0}: Error finding container 451d1c05cb8c8a01ad706231fb884a50320bbb0adaea5d857a109badd8364271: Status 404 returned error can't find the container with id 451d1c05cb8c8a01ad706231fb884a50320bbb0adaea5d857a109badd8364271
Sep 17 17:41:05.357289 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:05.356582    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8611822c_22f3_40b6_91fd_677ce0e5ae70.slice/crio-3c09956da39cbd4017b0c45783e9f63e88d3837311fa082db3c189c01118d17e WatchSource:0}: Error finding container 3c09956da39cbd4017b0c45783e9f63e88d3837311fa082db3c189c01118d17e: Status 404 returned error can't find the container with id 3c09956da39cbd4017b0c45783e9f63e88d3837311fa082db3c189c01118d17e
Sep 17 17:41:05.358443 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:05.357923    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54d9137_d225_4657_b601_b1a8bdee3b68.slice/crio-82d4698ffa21249127bed5a133855bb3bf13d11d87003fc6f065c194991efa07 WatchSource:0}: Error finding container 82d4698ffa21249127bed5a133855bb3bf13d11d87003fc6f065c194991efa07: Status 404 returned error can't find the container with id 82d4698ffa21249127bed5a133855bb3bf13d11d87003fc6f065c194991efa07
Sep 17 17:41:05.461022 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:05.460941    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:36:04 +0000 UTC" deadline="2028-05-03 00:04:23.319708631 +0000 UTC"
Sep 17 17:41:05.461022 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:05.461019    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14238h23m17.858693519s"
Sep 17 17:41:05.531898 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:05.531696    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:05.532050 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:05.532005    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7cdxt" podUID="a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2"
Sep 17 17:41:05.542121 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:05.542090    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rdmsd" event={"ID":"28657902-dde3-4d9b-bbaa-ce6abfefc4e9","Type":"ContainerStarted","Data":"8abc6b82a4e732604818b031f9dd098aab87b8cbd6ab5b64948e342f6990abda"}
Sep 17 17:41:05.543488 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:05.543462    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j5j7r" event={"ID":"8611822c-22f3-40b6-91fd-677ce0e5ae70","Type":"ContainerStarted","Data":"3c09956da39cbd4017b0c45783e9f63e88d3837311fa082db3c189c01118d17e"}
Sep 17 17:41:05.545376 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:05.545348    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7qls8" event={"ID":"0623a097-d7b1-4be4-be54-a29b81cc540a","Type":"ContainerStarted","Data":"451d1c05cb8c8a01ad706231fb884a50320bbb0adaea5d857a109badd8364271"}
Sep 17 17:41:05.547572 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:05.547536    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mb5dq" event={"ID":"feff5f74-e1df-4fdc-b2f3-ebd27908a886","Type":"ContainerStarted","Data":"78107dbba03764b1772d374421f026517605918a6f4d6b447106e90bac980911"}
Sep 17 17:41:05.548654 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:05.548626    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vmxsb" event={"ID":"d54d9137-d225-4657-b601-b1a8bdee3b68","Type":"ContainerStarted","Data":"82d4698ffa21249127bed5a133855bb3bf13d11d87003fc6f065c194991efa07"}
Sep 17 17:41:05.550835 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:05.550814    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7rmhq" event={"ID":"0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852","Type":"ContainerStarted","Data":"e36373421344352256402610de03757c27307c533ab94c493954a2701dcbde3e"}
Sep 17 17:41:05.552216 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:05.552193    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mh9jw" event={"ID":"8bacc5d4-bc80-46b0-93f5-c6dd572df3c1","Type":"ContainerStarted","Data":"b319992c0bca6b9fd4f05c512833e2fcb16dc0162f6c81d00eee61ba29e0406c"}
Sep 17 17:41:05.553343 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:05.553304    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf" event={"ID":"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b","Type":"ContainerStarted","Data":"7fceb6d232c3e319b074e14118cd72bdc147375cd630d5a3e11d652045a50d27"}
Sep 17 17:41:05.554548 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:05.554531    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp" event={"ID":"67858c18-1e78-4ca1-9ade-132a0e6251c7","Type":"ContainerStarted","Data":"e38a6bf605bb4ca17cd92d0529acc5bc93258a3f006211a1814bfda09ccd1419"}
Sep 17 17:41:05.556419 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:05.556403    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-127.ec2.internal" event={"ID":"dd6c8d1b0dcecbce259dac62f1441a28","Type":"ContainerStarted","Data":"900ceb535599d2d0e1c2d6d893903857bbce799c6648e4da9bce16e574d7335f"}
Sep 17 17:41:05.574208 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:05.574148    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-127.ec2.internal" podStartSLOduration=2.574129197 podStartE2EDuration="2.574129197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:05.573744563 +0000 UTC m=+5.166581543" watchObservedRunningTime="2026-09-17 17:41:05.574129197 +0000 UTC m=+5.166966174"
Sep 17 17:41:06.037624 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:06.036711    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret\") pod \"global-pull-secret-syncer-7cdxt\" (UID: \"a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2\") " pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:06.037624 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:06.036788    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll9v7\" (UniqueName: \"kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7\") pod \"network-check-target-qrrjr\" (UID: \"365faef8-3867-42a8-a227-d81293771f14\") " pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:06.037624 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:06.036964    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:06.037624 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:06.036985    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:06.037624 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:06.036998    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ll9v7 for pod openshift-network-diagnostics/network-check-target-qrrjr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:06.037624 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:06.037057    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7 podName:365faef8-3867-42a8-a227-d81293771f14 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:08.0370378 +0000 UTC m=+7.629874765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll9v7" (UniqueName: "kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7") pod "network-check-target-qrrjr" (UID: "365faef8-3867-42a8-a227-d81293771f14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:06.037624 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:06.037447    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:06.037624 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:06.037509    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret podName:a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:08.03748337 +0000 UTC m=+7.630320332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret") pod "global-pull-secret-syncer-7cdxt" (UID: "a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:06.138248 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:06.138209    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs\") pod \"network-metrics-daemon-z44z2\" (UID: \"84810a50-2551-4d78-ac6d-4d17f1524389\") " pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:06.138436 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:06.138365    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:06.138436 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:06.138433    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs podName:84810a50-2551-4d78-ac6d-4d17f1524389 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:08.138414171 +0000 UTC m=+7.731251146 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs") pod "network-metrics-daemon-z44z2" (UID: "84810a50-2551-4d78-ac6d-4d17f1524389") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:06.461858 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:06.461810    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:36:04 +0000 UTC" deadline="2028-05-06 14:18:40.388184475 +0000 UTC"
Sep 17 17:41:06.461858 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:06.461855    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14324h37m33.926333889s"
Sep 17 17:41:06.532232 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:06.532198    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:06.532405 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:06.532358    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z44z2" podUID="84810a50-2551-4d78-ac6d-4d17f1524389"
Sep 17 17:41:06.532768 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:06.532748    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:06.532893 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:06.532871    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrrjr" podUID="365faef8-3867-42a8-a227-d81293771f14"
Sep 17 17:41:06.536146 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:06.536122    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:41:06.573773 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:06.572512    2576 generic.go:356] "Generic (PLEG): container finished" podID="e61ef62b30beabff4e4bbd49cb7c73f3" containerID="9008d1ed76cca018436cd1143151410205fb67a0ec3b2d258a2c6566af1c8b19" exitCode=0
Sep 17 17:41:06.573773 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:06.573482    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal" event={"ID":"e61ef62b30beabff4e4bbd49cb7c73f3","Type":"ContainerDied","Data":"9008d1ed76cca018436cd1143151410205fb67a0ec3b2d258a2c6566af1c8b19"}
Sep 17 17:41:07.531845 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:07.531797    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:07.532363 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:07.531951    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7cdxt" podUID="a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2"
Sep 17 17:41:07.590775 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:07.590699    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal" event={"ID":"e61ef62b30beabff4e4bbd49cb7c73f3","Type":"ContainerStarted","Data":"f3ebe51ad8db2e540beec94458e59e24d562fedbf8955d807d73cc6c6212f0b8"}
Sep 17 17:41:08.056942 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:08.056899    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret\") pod \"global-pull-secret-syncer-7cdxt\" (UID: \"a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2\") " pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:08.057124 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:08.056971    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll9v7\" (UniqueName: \"kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7\") pod \"network-check-target-qrrjr\" (UID: \"365faef8-3867-42a8-a227-d81293771f14\") " pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:08.057124 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:08.057106    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:08.057124 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:08.057122    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:08.057295 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:08.057135    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ll9v7 for pod openshift-network-diagnostics/network-check-target-qrrjr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:08.057295 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:08.057193    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7 podName:365faef8-3867-42a8-a227-d81293771f14 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:12.057174281 +0000 UTC m=+11.650011245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll9v7" (UniqueName: "kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7") pod "network-check-target-qrrjr" (UID: "365faef8-3867-42a8-a227-d81293771f14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:08.057605 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:08.057587    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:08.057697 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:08.057644    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret podName:a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:12.057628876 +0000 UTC m=+11.650465850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret") pod "global-pull-secret-syncer-7cdxt" (UID: "a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:08.157337 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:08.157298    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs\") pod \"network-metrics-daemon-z44z2\" (UID: \"84810a50-2551-4d78-ac6d-4d17f1524389\") " pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:08.157527 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:08.157510    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:08.157604 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:08.157597    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs podName:84810a50-2551-4d78-ac6d-4d17f1524389 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:12.157578373 +0000 UTC m=+11.750415347 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs") pod "network-metrics-daemon-z44z2" (UID: "84810a50-2551-4d78-ac6d-4d17f1524389") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:08.537082 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:08.536367    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:08.537082 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:08.536515    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z44z2" podUID="84810a50-2551-4d78-ac6d-4d17f1524389"
Sep 17 17:41:08.537082 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:08.536954    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:08.537082 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:08.537039    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrrjr" podUID="365faef8-3867-42a8-a227-d81293771f14"
Sep 17 17:41:09.532510 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:09.532468    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:09.532699 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:09.532628    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7cdxt" podUID="a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2"
Sep 17 17:41:10.531993 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:10.531925    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:10.532478 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:10.532071    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrrjr" podUID="365faef8-3867-42a8-a227-d81293771f14"
Sep 17 17:41:10.532478 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:10.532102    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:10.532478 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:10.532218    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z44z2" podUID="84810a50-2551-4d78-ac6d-4d17f1524389"
Sep 17 17:41:11.532132 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:11.532093    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:11.532645 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:11.532246    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7cdxt" podUID="a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2"
Sep 17 17:41:12.091125 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:12.090693    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret\") pod \"global-pull-secret-syncer-7cdxt\" (UID: \"a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2\") " pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:12.091125 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:12.090797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll9v7\" (UniqueName: \"kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7\") pod \"network-check-target-qrrjr\" (UID: \"365faef8-3867-42a8-a227-d81293771f14\") " pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:12.091125 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:12.090846    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:12.091125 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:12.090940    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret podName:a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:20.090904739 +0000 UTC m=+19.683741724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret") pod "global-pull-secret-syncer-7cdxt" (UID: "a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:12.091125 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:12.090974    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:12.091125 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:12.090995    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:12.091125 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:12.091008    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ll9v7 for pod openshift-network-diagnostics/network-check-target-qrrjr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:12.091125 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:12.091092    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7 podName:365faef8-3867-42a8-a227-d81293771f14 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:20.091073758 +0000 UTC m=+19.683910737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll9v7" (UniqueName: "kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7") pod "network-check-target-qrrjr" (UID: "365faef8-3867-42a8-a227-d81293771f14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:12.192049 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:12.192004    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs\") pod \"network-metrics-daemon-z44z2\" (UID: \"84810a50-2551-4d78-ac6d-4d17f1524389\") " pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:12.192231 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:12.192138    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:12.192231 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:12.192202    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs podName:84810a50-2551-4d78-ac6d-4d17f1524389 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:20.192184616 +0000 UTC m=+19.785021580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs") pod "network-metrics-daemon-z44z2" (UID: "84810a50-2551-4d78-ac6d-4d17f1524389") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:12.536150 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:12.536064    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:12.536598 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:12.536200    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z44z2" podUID="84810a50-2551-4d78-ac6d-4d17f1524389"
Sep 17 17:41:12.536702 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:12.536551    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:12.536763 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:12.536736    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrrjr" podUID="365faef8-3867-42a8-a227-d81293771f14"
Sep 17 17:41:13.532428 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:13.532337    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:13.532642 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:13.532475    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7cdxt" podUID="a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2"
Sep 17 17:41:14.535361 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:14.535334    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:14.535361 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:14.535345    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:14.535824 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:14.535452    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrrjr" podUID="365faef8-3867-42a8-a227-d81293771f14"
Sep 17 17:41:14.535824 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:14.535593    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z44z2" podUID="84810a50-2551-4d78-ac6d-4d17f1524389"
Sep 17 17:41:15.532329 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:15.532241    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:15.532601 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:15.532396    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7cdxt" podUID="a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2"
Sep 17 17:41:16.532354 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:16.532321    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:16.532821 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:16.532322    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:16.532821 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:16.532463    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z44z2" podUID="84810a50-2551-4d78-ac6d-4d17f1524389"
Sep 17 17:41:16.532821 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:16.532520    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrrjr" podUID="365faef8-3867-42a8-a227-d81293771f14"
Sep 17 17:41:17.531749 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:17.531711    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:17.531924 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:17.531820    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7cdxt" podUID="a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2"
Sep 17 17:41:18.532025 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:18.531990    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:18.532429 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:18.531918    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:18.532429 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:18.532130    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z44z2" podUID="84810a50-2551-4d78-ac6d-4d17f1524389"
Sep 17 17:41:18.532429 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:18.532246    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrrjr" podUID="365faef8-3867-42a8-a227-d81293771f14"
Sep 17 17:41:19.532314 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:19.532282    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:19.532807 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:19.532410    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7cdxt" podUID="a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2"
Sep 17 17:41:20.150278 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:20.150240    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret\") pod \"global-pull-secret-syncer-7cdxt\" (UID: \"a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2\") " pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:20.150471 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:20.150313    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll9v7\" (UniqueName: \"kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7\") pod \"network-check-target-qrrjr\" (UID: \"365faef8-3867-42a8-a227-d81293771f14\") " pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:20.150471 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:20.150394    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:20.150471 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:20.150427    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:20.150471 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:20.150447    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:20.150471 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:20.150462    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ll9v7 for pod openshift-network-diagnostics/network-check-target-qrrjr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:20.150725 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:20.150477    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret podName:a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:36.150452885 +0000 UTC m=+35.743289846 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret") pod "global-pull-secret-syncer-7cdxt" (UID: "a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:20.150725 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:20.150510    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7 podName:365faef8-3867-42a8-a227-d81293771f14 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:36.150494653 +0000 UTC m=+35.743331614 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll9v7" (UniqueName: "kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7") pod "network-check-target-qrrjr" (UID: "365faef8-3867-42a8-a227-d81293771f14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:20.250935 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:20.250888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs\") pod \"network-metrics-daemon-z44z2\" (UID: \"84810a50-2551-4d78-ac6d-4d17f1524389\") " pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:20.251088 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:20.251046    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:20.251134 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:20.251106    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs podName:84810a50-2551-4d78-ac6d-4d17f1524389 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:36.251091118 +0000 UTC m=+35.843928078 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs") pod "network-metrics-daemon-z44z2" (UID: "84810a50-2551-4d78-ac6d-4d17f1524389") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:20.531904 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:20.531817    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:20.532074 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:20.531816    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:20.532074 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:20.531971    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z44z2" podUID="84810a50-2551-4d78-ac6d-4d17f1524389"
Sep 17 17:41:20.532074 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:20.532040    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrrjr" podUID="365faef8-3867-42a8-a227-d81293771f14"
Sep 17 17:41:21.532543 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:21.532511    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:21.532998 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:21.532633    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7cdxt" podUID="a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2"
Sep 17 17:41:22.532997 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:22.532971    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:22.533349 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:22.533096    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z44z2" podUID="84810a50-2551-4d78-ac6d-4d17f1524389"
Sep 17 17:41:22.533349 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:22.533144    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:22.533349 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:22.533258    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrrjr" podUID="365faef8-3867-42a8-a227-d81293771f14"
Sep 17 17:41:22.624681 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:22.624651    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp" event={"ID":"67858c18-1e78-4ca1-9ade-132a0e6251c7","Type":"ContainerStarted","Data":"6653f6cea0c17b6fa49a9373fc0536266bef4b0935e8287591164c330cb9a2e8"}
Sep 17 17:41:22.626243 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:22.626218    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rdmsd" event={"ID":"28657902-dde3-4d9b-bbaa-ce6abfefc4e9","Type":"ContainerStarted","Data":"325a10ea93a80ee1f5db095155529bbcba56bdfea8553cc80047321b99e115a0"}
Sep 17 17:41:22.628085 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:22.628056    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j5j7r" event={"ID":"8611822c-22f3-40b6-91fd-677ce0e5ae70","Type":"ContainerStarted","Data":"5cadddd7d1013b5dd1c7e652151185f4eb21d85666d451d5ed0af55644bb4f3d"}
Sep 17 17:41:22.630294 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:22.630272    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7qls8" event={"ID":"0623a097-d7b1-4be4-be54-a29b81cc540a","Type":"ContainerStarted","Data":"cfb3923aec2c1ce0645e49efbadd25f6fc00ca56d37625962daa9f0ce6698fcd"}
Sep 17 17:41:22.631610 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:22.631589    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mb5dq" event={"ID":"feff5f74-e1df-4fdc-b2f3-ebd27908a886","Type":"ContainerStarted","Data":"89f57432f0f5794fbe6912a64d9ffe88c8561f39dc5965ec9f000af9de225047"}
Sep 17 17:41:22.632954 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:22.632934    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vmxsb" event={"ID":"d54d9137-d225-4657-b601-b1a8bdee3b68","Type":"ContainerStarted","Data":"a53b6d2f7e435c87ae4c6440005dd46342c6e2916acb9bc8d991ebb2577b9e68"}
Sep 17 17:41:22.634289 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:22.634270    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7rmhq" event={"ID":"0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852","Type":"ContainerStarted","Data":"751af7b82ec9b7a4a8fd89f497ac1b871c7dbb44f8c23eadef5c6c06b378a2a2"}
Sep 17 17:41:22.654399 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:22.654321    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-127.ec2.internal" podStartSLOduration=19.654307941 podStartE2EDuration="19.654307941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:07.607661374 +0000 UTC m=+7.200498356" watchObservedRunningTime="2026-09-17 17:41:22.654307941 +0000 UTC m=+22.247144923"
Sep 17 17:41:22.679050 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:22.679014    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rdmsd" podStartSLOduration=3.802982935 podStartE2EDuration="20.67899988s" totalImagesPullingTime="16.876016945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:22.654461462 +0000 UTC m=+22.247298425" watchObservedRunningTime="2026-09-17 17:41:22.67899988 +0000 UTC m=+22.271836861"
Sep 17 17:41:22.700656 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:22.700536    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mb5dq" podStartSLOduration=3.908091374 podStartE2EDuration="20.700511612s" totalImagesPullingTime="16.792420238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:22.696085512 +0000 UTC m=+22.288922590" watchObservedRunningTime="2026-09-17 17:41:22.700511612 +0000 UTC m=+22.293348593"
Sep 17 17:41:22.712961 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:22.712922    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7rmhq" podStartSLOduration=3.855848926 podStartE2EDuration="20.712907508s" totalImagesPullingTime="16.857058582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:22.712869795 +0000 UTC m=+22.305706775" watchObservedRunningTime="2026-09-17 17:41:22.712907508 +0000 UTC m=+22.305744472"
Sep 17 17:41:22.735875 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:22.735835    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vmxsb" podStartSLOduration=3.883585048 podStartE2EDuration="20.735820665s" totalImagesPullingTime="16.852235617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:22.73546918 +0000 UTC m=+22.328306160" watchObservedRunningTime="2026-09-17 17:41:22.735820665 +0000 UTC m=+22.328657645"
Sep 17 17:41:22.760840 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:22.760799    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j5j7r" podStartSLOduration=3.90568776 podStartE2EDuration="20.760782001s" totalImagesPullingTime="16.855094241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:22.760058328 +0000 UTC m=+22.352895309" watchObservedRunningTime="2026-09-17 17:41:22.760782001 +0000 UTC m=+22.353618982"
Sep 17 17:41:23.478236 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:23.478087    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:41:23.502440 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:23.502360    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:41:23.478230334Z","UUID":"3bbae44b-d623-4de3-999b-287616f7b27b","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:41:23.503684 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:23.503667    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:41:23.503771 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:23.503697    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:41:23.532412 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:23.532391    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:23.532512 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:23.532479    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7cdxt" podUID="a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2"
Sep 17 17:41:23.637827 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:23.637793    2576 generic.go:356] "Generic (PLEG): container finished" podID="0623a097-d7b1-4be4-be54-a29b81cc540a" containerID="cfb3923aec2c1ce0645e49efbadd25f6fc00ca56d37625962daa9f0ce6698fcd" exitCode=0
Sep 17 17:41:23.638706 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:23.637881    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7qls8" event={"ID":"0623a097-d7b1-4be4-be54-a29b81cc540a","Type":"ContainerDied","Data":"cfb3923aec2c1ce0645e49efbadd25f6fc00ca56d37625962daa9f0ce6698fcd"}
Sep 17 17:41:23.640985 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:23.640955    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/ovn-acl-logging/0.log"
Sep 17 17:41:23.641367 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:23.641345    2576 generic.go:356] "Generic (PLEG): container finished" podID="5bc15f81-98ec-416f-a1bd-9f5d6d2a367b" containerID="083e68aaf281e458fe4cd19fc34fd93ff759a3daa63815ee8e1234332ef2f4f9" exitCode=1
Sep 17 17:41:23.641440 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:23.641421    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf" event={"ID":"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b","Type":"ContainerStarted","Data":"5ffd935935938a3ae4ce9a0b50d9c89376345e509feade8d9fcca7d503caf50a"}
Sep 17 17:41:23.641510 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:23.641491    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf" event={"ID":"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b","Type":"ContainerStarted","Data":"37c1884b76d4e23c5b27fefb6994d75f18e00ac2cd02c52a40e2ac29a0220c4a"}
Sep 17 17:41:23.641551 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:23.641539    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf" event={"ID":"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b","Type":"ContainerStarted","Data":"a813f35a4c402bb97048e1219dfa0fd8801947003b658fffd4641776b40f7bd1"}
Sep 17 17:41:23.641615 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:23.641602    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf" event={"ID":"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b","Type":"ContainerStarted","Data":"bc29e10c67c38c124d4d088ea2701edbb45cd5b15eef9cf9a9bb9f2572a47ec0"}
Sep 17 17:41:23.641664 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:23.641651    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf" event={"ID":"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b","Type":"ContainerStarted","Data":"ccd2540c7f38e013a9627f6a13579dfd94c90d501098a766ae1ff17052699bd9"}
Sep 17 17:41:23.641732 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:23.641720    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf" event={"ID":"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b","Type":"ContainerDied","Data":"083e68aaf281e458fe4cd19fc34fd93ff759a3daa63815ee8e1234332ef2f4f9"}
Sep 17 17:41:23.643115 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:23.643097    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp" event={"ID":"67858c18-1e78-4ca1-9ade-132a0e6251c7","Type":"ContainerStarted","Data":"fc1021dd954bf00f8b4ad7293a05e3ece5d2be9b0ce04bfef78b35745c4d983e"}
Sep 17 17:41:24.532730 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:24.532653    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:24.532880 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:24.532764    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrrjr" podUID="365faef8-3867-42a8-a227-d81293771f14"
Sep 17 17:41:24.532880 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:24.532869    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:24.533006 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:24.532987    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z44z2" podUID="84810a50-2551-4d78-ac6d-4d17f1524389"
Sep 17 17:41:24.647500 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:24.647436    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mh9jw" event={"ID":"8bacc5d4-bc80-46b0-93f5-c6dd572df3c1","Type":"ContainerStarted","Data":"30476000dfc39722c36f86323b44355cf9a421c46bc670622d7c5cf01b29f4e5"}
Sep 17 17:41:24.668963 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:24.668916    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mh9jw" podStartSLOduration=5.803228247 podStartE2EDuration="22.66889761s" totalImagesPullingTime="16.865669363s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:24.668839211 +0000 UTC m=+24.261676192" watchObservedRunningTime="2026-09-17 17:41:24.66889761 +0000 UTC m=+24.261734594"
Sep 17 17:41:24.975046 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:24.975008    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mb5dq"
Sep 17 17:41:24.975683 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:24.975661    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mb5dq"
Sep 17 17:41:25.532174 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:25.532145    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:25.532341 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:25.532266    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7cdxt" podUID="a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2"
Sep 17 17:41:25.654177 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:25.654146    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/ovn-acl-logging/0.log"
Sep 17 17:41:25.654687 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:25.654649    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf" event={"ID":"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b","Type":"ContainerStarted","Data":"aa2e3db05ee3f4e1e6d5ff85401df6c92791caf2abbc0cb7afe7e06a8a93b20a"}
Sep 17 17:41:25.657013 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:25.656949    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp" event={"ID":"67858c18-1e78-4ca1-9ade-132a0e6251c7","Type":"ContainerStarted","Data":"5e7c1cba7fb848a2dd418f5c93eabbdded1d07d15272a262287fc8992d63b168"}
Sep 17 17:41:25.657334 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:25.657283    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mb5dq"
Sep 17 17:41:25.657955 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:25.657939    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mb5dq"
Sep 17 17:41:25.707790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:25.707734    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npsqp" podStartSLOduration=4.9278691519999995 podStartE2EDuration="23.70771225s" totalImagesPullingTime="18.779843098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:25.707112571 +0000 UTC m=+25.299949551" watchObservedRunningTime="2026-09-17 17:41:25.70771225 +0000 UTC m=+25.300549231"
Sep 17 17:41:26.532361 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:26.532163    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:26.532598 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:26.532222    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:26.532598 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:26.532476    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z44z2" podUID="84810a50-2551-4d78-ac6d-4d17f1524389"
Sep 17 17:41:26.532598 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:26.532544    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrrjr" podUID="365faef8-3867-42a8-a227-d81293771f14"
Sep 17 17:41:27.532518 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:27.532496    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:27.532999 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:27.532604    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7cdxt" podUID="a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2"
Sep 17 17:41:27.666088 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:27.665883    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/ovn-acl-logging/0.log"
Sep 17 17:41:27.666542 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:27.666509    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf" event={"ID":"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b","Type":"ContainerStarted","Data":"876f34ce33c36e85fab7ebe3746988a1ffe9795d2db3e57ba64e0cdebb0157d1"}
Sep 17 17:41:27.666913 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:27.666876    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:27.667141 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:27.667119    2576 scope.go:117] "RemoveContainer" containerID="083e68aaf281e458fe4cd19fc34fd93ff759a3daa63815ee8e1234332ef2f4f9"
Sep 17 17:41:27.669203 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:27.669179    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7qls8" event={"ID":"0623a097-d7b1-4be4-be54-a29b81cc540a","Type":"ContainerStarted","Data":"1a7804f16633d9f26b6eb5d0aa2d92e1cc84253f0b399908eca6bd5006be81cc"}
Sep 17 17:41:27.683352 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:27.683329    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:28.532352 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:28.532310    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:28.532530 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:28.532322    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:28.532530 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:28.532428    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z44z2" podUID="84810a50-2551-4d78-ac6d-4d17f1524389"
Sep 17 17:41:28.533105 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:28.533066    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrrjr" podUID="365faef8-3867-42a8-a227-d81293771f14"
Sep 17 17:41:28.673170 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:28.673139    2576 generic.go:356] "Generic (PLEG): container finished" podID="0623a097-d7b1-4be4-be54-a29b81cc540a" containerID="1a7804f16633d9f26b6eb5d0aa2d92e1cc84253f0b399908eca6bd5006be81cc" exitCode=0
Sep 17 17:41:28.673332 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:28.673228    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7qls8" event={"ID":"0623a097-d7b1-4be4-be54-a29b81cc540a","Type":"ContainerDied","Data":"1a7804f16633d9f26b6eb5d0aa2d92e1cc84253f0b399908eca6bd5006be81cc"}
Sep 17 17:41:28.677137 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:28.677121    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/ovn-acl-logging/0.log"
Sep 17 17:41:28.677468 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:28.677450    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf" event={"ID":"5bc15f81-98ec-416f-a1bd-9f5d6d2a367b","Type":"ContainerStarted","Data":"a1123e9e23543c4d3bfb347f50f220c65810f6d520996374f17de4dced7f0224"}
Sep 17 17:41:28.677582 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:28.677556    2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:41:28.677794 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:28.677771    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:28.692597 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:28.692579    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:28.728787 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:28.728743    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf" podStartSLOduration=9.539943427 podStartE2EDuration="26.728725722s" totalImagesPullingTime="17.188782295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:28.728258227 +0000 UTC m=+28.321095207" watchObservedRunningTime="2026-09-17 17:41:28.728725722 +0000 UTC m=+28.321562702"
Sep 17 17:41:29.463681 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:29.463620    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7cdxt"]
Sep 17 17:41:29.463817 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:29.463746    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:29.463880 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:29.463856    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7cdxt" podUID="a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2"
Sep 17 17:41:29.464295 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:29.464263    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z44z2"]
Sep 17 17:41:29.464407 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:29.464370    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:29.464491 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:29.464469    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z44z2" podUID="84810a50-2551-4d78-ac6d-4d17f1524389"
Sep 17 17:41:29.464994 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:29.464970    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qrrjr"]
Sep 17 17:41:29.465073 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:29.465053    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:29.465136 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:29.465116    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrrjr" podUID="365faef8-3867-42a8-a227-d81293771f14"
Sep 17 17:41:29.682108 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:29.682076    2576 generic.go:356] "Generic (PLEG): container finished" podID="0623a097-d7b1-4be4-be54-a29b81cc540a" containerID="399ee5cd2937a5ccc9b6531510a7a6f7f50843f2455186e27be37fc0f1b6ea85" exitCode=0
Sep 17 17:41:29.682534 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:29.682151    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7qls8" event={"ID":"0623a097-d7b1-4be4-be54-a29b81cc540a","Type":"ContainerDied","Data":"399ee5cd2937a5ccc9b6531510a7a6f7f50843f2455186e27be37fc0f1b6ea85"}
Sep 17 17:41:29.682534 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:29.682249    2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:41:30.224352 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:30.224155    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:41:30.532194 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:30.532107    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:30.532330 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:30.532250    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrrjr" podUID="365faef8-3867-42a8-a227-d81293771f14"
Sep 17 17:41:30.686513 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:30.686481    2576 generic.go:356] "Generic (PLEG): container finished" podID="0623a097-d7b1-4be4-be54-a29b81cc540a" containerID="9d8819e1d5ca34bc6b23a0d96ce6b83239694623c3951e313d320915d0b9919b" exitCode=0
Sep 17 17:41:30.686878 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:30.686582    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7qls8" event={"ID":"0623a097-d7b1-4be4-be54-a29b81cc540a","Type":"ContainerDied","Data":"9d8819e1d5ca34bc6b23a0d96ce6b83239694623c3951e313d320915d0b9919b"}
Sep 17 17:41:31.532360 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:31.532327    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:31.532533 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:31.532371    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:31.532533 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:31.532462    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z44z2" podUID="84810a50-2551-4d78-ac6d-4d17f1524389"
Sep 17 17:41:31.532671 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:31.532602    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7cdxt" podUID="a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2"
Sep 17 17:41:32.532552 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:32.532478    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:32.533059 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:32.532600    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrrjr" podUID="365faef8-3867-42a8-a227-d81293771f14"
Sep 17 17:41:33.532633 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:33.532595    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:33.533019 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:33.532596    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:33.533019 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:33.532737    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7cdxt" podUID="a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2"
Sep 17 17:41:33.533019 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:33.532791    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z44z2" podUID="84810a50-2551-4d78-ac6d-4d17f1524389"
Sep 17 17:41:34.532355 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:34.532276    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:34.532500 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:34.532400    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrrjr" podUID="365faef8-3867-42a8-a227-d81293771f14"
Sep 17 17:41:35.531921 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:35.531894    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:35.532482 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:35.531898    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:35.532482 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:35.532087    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z44z2" podUID="84810a50-2551-4d78-ac6d-4d17f1524389"
Sep 17 17:41:35.532482 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:35.531985    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7cdxt" podUID="a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2"
Sep 17 17:41:35.704359 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:35.704324    2576 generic.go:356] "Generic (PLEG): container finished" podID="0623a097-d7b1-4be4-be54-a29b81cc540a" containerID="eca4807836ba109ec7d7efb9f86b3907b284c99dd767f31b2a31afe0dcda69c2" exitCode=0
Sep 17 17:41:35.704531 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:35.704377    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7qls8" event={"ID":"0623a097-d7b1-4be4-be54-a29b81cc540a","Type":"ContainerDied","Data":"eca4807836ba109ec7d7efb9f86b3907b284c99dd767f31b2a31afe0dcda69c2"}
Sep 17 17:41:36.156856 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.156819    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll9v7\" (UniqueName: \"kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7\") pod \"network-check-target-qrrjr\" (UID: \"365faef8-3867-42a8-a227-d81293771f14\") " pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:36.156993 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.156882    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret\") pod \"global-pull-secret-syncer-7cdxt\" (UID: \"a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2\") " pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:36.156993 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:36.156973    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:36.157096 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:36.157004    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:41:36.157096 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:36.157024    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret podName:a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:08.157010327 +0000 UTC m=+67.749847287 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret") pod "global-pull-secret-syncer-7cdxt" (UID: "a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:41:36.157096 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:36.157033    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:41:36.157096 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:36.157047    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ll9v7 for pod openshift-network-diagnostics/network-check-target-qrrjr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:36.157239 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:36.157098    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7 podName:365faef8-3867-42a8-a227-d81293771f14 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:08.157081646 +0000 UTC m=+67.749918618 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ll9v7" (UniqueName: "kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7") pod "network-check-target-qrrjr" (UID: "365faef8-3867-42a8-a227-d81293771f14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:41:36.257418 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.257384    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs\") pod \"network-metrics-daemon-z44z2\" (UID: \"84810a50-2551-4d78-ac6d-4d17f1524389\") " pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:36.257602 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:36.257490    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:36.257602 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:36.257541    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs podName:84810a50-2551-4d78-ac6d-4d17f1524389 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:08.257529258 +0000 UTC m=+67.850366230 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs") pod "network-metrics-daemon-z44z2" (UID: "84810a50-2551-4d78-ac6d-4d17f1524389") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:41:36.296668 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.296640    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-127.ec2.internal" event="NodeReady"
Sep 17 17:41:36.296810 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.296769    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:41:36.347973 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.347946    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qjlh9"]
Sep 17 17:41:36.350122 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.350102    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67458c6fcd-jkctp"]
Sep 17 17:41:36.350245 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.350213    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:36.351769 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.351747    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5zhfj"]
Sep 17 17:41:36.351903 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.351872    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.352637 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.352618    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:41:36.352637 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.352632    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fglck\""
Sep 17 17:41:36.353105 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.353086    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:41:36.353436 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.353422    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5zhfj"
Sep 17 17:41:36.355356 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.355340    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mtf7c\""
Sep 17 17:41:36.356299 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.356277    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:41:36.356497 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.356451    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:41:36.356497 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.356465    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:41:36.356645 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.356523    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:41:36.357260 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.357243    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:41:36.357364 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.357351    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kcr8x\""
Sep 17 17:41:36.357758 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.357531    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:41:36.360905 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.360886    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:41:36.364091 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.364073    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5zhfj"]
Sep 17 17:41:36.367670 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.367653    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qjlh9"]
Sep 17 17:41:36.368450 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.368429    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67458c6fcd-jkctp"]
Sep 17 17:41:36.459034 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.458999    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2353bb3f-e217-4fb7-9202-8c567d93cf29-cert\") pod \"ingress-canary-5zhfj\" (UID: \"2353bb3f-e217-4fb7-9202-8c567d93cf29\") " pod="openshift-ingress-canary/ingress-canary-5zhfj"
Sep 17 17:41:36.459034 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.459038    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f80b68ad-44e6-40fd-828b-aa73ab059235-config-volume\") pod \"dns-default-qjlh9\" (UID: \"f80b68ad-44e6-40fd-828b-aa73ab059235\") " pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:36.459283 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.459067    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/724b2fc7-ecc7-4b88-98ff-79188ad6d792-image-registry-private-configuration\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.459283 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.459110    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-certificates\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.459283 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.459161    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-bound-sa-token\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.459283 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.459188    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f80b68ad-44e6-40fd-828b-aa73ab059235-metrics-tls\") pod \"dns-default-qjlh9\" (UID: \"f80b68ad-44e6-40fd-828b-aa73ab059235\") " pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:36.459283 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.459262    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.459442 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.459291    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/724b2fc7-ecc7-4b88-98ff-79188ad6d792-ca-trust-extracted\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.459442 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.459305    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80b68ad-44e6-40fd-828b-aa73ab059235-tmp-dir\") pod \"dns-default-qjlh9\" (UID: \"f80b68ad-44e6-40fd-828b-aa73ab059235\") " pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:36.459442 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.459338    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/724b2fc7-ecc7-4b88-98ff-79188ad6d792-trusted-ca\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.459442 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.459363    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/724b2fc7-ecc7-4b88-98ff-79188ad6d792-installation-pull-secrets\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.459442 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.459377    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfqpw\" (UniqueName: \"kubernetes.io/projected/f80b68ad-44e6-40fd-828b-aa73ab059235-kube-api-access-sfqpw\") pod \"dns-default-qjlh9\" (UID: \"f80b68ad-44e6-40fd-828b-aa73ab059235\") " pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:36.459442 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.459414    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt9r5\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-kube-api-access-mt9r5\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.459442 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.459437    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6h4l\" (UniqueName: \"kubernetes.io/projected/2353bb3f-e217-4fb7-9202-8c567d93cf29-kube-api-access-n6h4l\") pod \"ingress-canary-5zhfj\" (UID: \"2353bb3f-e217-4fb7-9202-8c567d93cf29\") " pod="openshift-ingress-canary/ingress-canary-5zhfj"
Sep 17 17:41:36.531910 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.531878    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:41:36.534533 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.534509    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hq5mj\""
Sep 17 17:41:36.534533 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.534526    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:41:36.535016 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.534509    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:41:36.560623 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.560598    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt9r5\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-kube-api-access-mt9r5\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.560725 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.560627    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6h4l\" (UniqueName: \"kubernetes.io/projected/2353bb3f-e217-4fb7-9202-8c567d93cf29-kube-api-access-n6h4l\") pod \"ingress-canary-5zhfj\" (UID: \"2353bb3f-e217-4fb7-9202-8c567d93cf29\") " pod="openshift-ingress-canary/ingress-canary-5zhfj"
Sep 17 17:41:36.560779 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.560762    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2353bb3f-e217-4fb7-9202-8c567d93cf29-cert\") pod \"ingress-canary-5zhfj\" (UID: \"2353bb3f-e217-4fb7-9202-8c567d93cf29\") " pod="openshift-ingress-canary/ingress-canary-5zhfj"
Sep 17 17:41:36.560829 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.560804    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f80b68ad-44e6-40fd-828b-aa73ab059235-config-volume\") pod \"dns-default-qjlh9\" (UID: \"f80b68ad-44e6-40fd-828b-aa73ab059235\") " pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:36.560829 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.560824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/724b2fc7-ecc7-4b88-98ff-79188ad6d792-image-registry-private-configuration\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.560918 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.560841    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-certificates\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.560918 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.560862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-bound-sa-token\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.565420 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.561371    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f80b68ad-44e6-40fd-828b-aa73ab059235-metrics-tls\") pod \"dns-default-qjlh9\" (UID: \"f80b68ad-44e6-40fd-828b-aa73ab059235\") " pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:36.565420 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.561474    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.565420 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.561507    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/724b2fc7-ecc7-4b88-98ff-79188ad6d792-ca-trust-extracted\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.565420 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.561537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80b68ad-44e6-40fd-828b-aa73ab059235-tmp-dir\") pod \"dns-default-qjlh9\" (UID: \"f80b68ad-44e6-40fd-828b-aa73ab059235\") " pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:36.565420 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.561600    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/724b2fc7-ecc7-4b88-98ff-79188ad6d792-trusted-ca\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.565420 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.561634    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/724b2fc7-ecc7-4b88-98ff-79188ad6d792-installation-pull-secrets\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.565420 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.561672    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfqpw\" (UniqueName: \"kubernetes.io/projected/f80b68ad-44e6-40fd-828b-aa73ab059235-kube-api-access-sfqpw\") pod \"dns-default-qjlh9\" (UID: \"f80b68ad-44e6-40fd-828b-aa73ab059235\") " pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:36.565420 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.561834    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-certificates\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.565420 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.561853    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f80b68ad-44e6-40fd-828b-aa73ab059235-config-volume\") pod \"dns-default-qjlh9\" (UID: \"f80b68ad-44e6-40fd-828b-aa73ab059235\") " pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:36.565420 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:36.561968    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:41:36.565420 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:36.561983    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67458c6fcd-jkctp: secret "image-registry-tls" not found
Sep 17 17:41:36.565420 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.562023    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80b68ad-44e6-40fd-828b-aa73ab059235-tmp-dir\") pod \"dns-default-qjlh9\" (UID: \"f80b68ad-44e6-40fd-828b-aa73ab059235\") " pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:36.565420 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:36.562040    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls podName:724b2fc7-ecc7-4b88-98ff-79188ad6d792 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:37.062022399 +0000 UTC m=+36.654859366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls") pod "image-registry-67458c6fcd-jkctp" (UID: "724b2fc7-ecc7-4b88-98ff-79188ad6d792") : secret "image-registry-tls" not found
Sep 17 17:41:36.565420 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.562439    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/724b2fc7-ecc7-4b88-98ff-79188ad6d792-ca-trust-extracted\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.565420 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:36.562735    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:36.565420 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:36.562792    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f80b68ad-44e6-40fd-828b-aa73ab059235-metrics-tls podName:f80b68ad-44e6-40fd-828b-aa73ab059235 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:37.062776382 +0000 UTC m=+36.655613342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f80b68ad-44e6-40fd-828b-aa73ab059235-metrics-tls") pod "dns-default-qjlh9" (UID: "f80b68ad-44e6-40fd-828b-aa73ab059235") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:36.565420 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.565082    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/724b2fc7-ecc7-4b88-98ff-79188ad6d792-trusted-ca\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.566420 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.566397    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/724b2fc7-ecc7-4b88-98ff-79188ad6d792-image-registry-private-configuration\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.566930 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.566899    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/724b2fc7-ecc7-4b88-98ff-79188ad6d792-installation-pull-secrets\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.567296 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.567280    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2353bb3f-e217-4fb7-9202-8c567d93cf29-cert\") pod \"ingress-canary-5zhfj\" (UID: \"2353bb3f-e217-4fb7-9202-8c567d93cf29\") " pod="openshift-ingress-canary/ingress-canary-5zhfj"
Sep 17 17:41:36.573244 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.573216    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-bound-sa-token\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.575200 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.575176    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6h4l\" (UniqueName: \"kubernetes.io/projected/2353bb3f-e217-4fb7-9202-8c567d93cf29-kube-api-access-n6h4l\") pod \"ingress-canary-5zhfj\" (UID: \"2353bb3f-e217-4fb7-9202-8c567d93cf29\") " pod="openshift-ingress-canary/ingress-canary-5zhfj"
Sep 17 17:41:36.575311 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.575290    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt9r5\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-kube-api-access-mt9r5\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:36.575841 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.575822    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfqpw\" (UniqueName: \"kubernetes.io/projected/f80b68ad-44e6-40fd-828b-aa73ab059235-kube-api-access-sfqpw\") pod \"dns-default-qjlh9\" (UID: \"f80b68ad-44e6-40fd-828b-aa73ab059235\") " pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:36.673263 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.673192    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5zhfj"
Sep 17 17:41:36.710078 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.710043    2576 generic.go:356] "Generic (PLEG): container finished" podID="0623a097-d7b1-4be4-be54-a29b81cc540a" containerID="89ccccb691c7c5516b40790fccaa219a27318538baeb2bc621a3d54f7fabc185" exitCode=0
Sep 17 17:41:36.710234 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.710094    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7qls8" event={"ID":"0623a097-d7b1-4be4-be54-a29b81cc540a","Type":"ContainerDied","Data":"89ccccb691c7c5516b40790fccaa219a27318538baeb2bc621a3d54f7fabc185"}
Sep 17 17:41:36.953552 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:36.953267    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5zhfj"]
Sep 17 17:41:36.958051 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:36.958016    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2353bb3f_e217_4fb7_9202_8c567d93cf29.slice/crio-e48c7e2fc75cc5592f34c3e20818491cd73a3b92580fdef0b5acae67fef594b1 WatchSource:0}: Error finding container e48c7e2fc75cc5592f34c3e20818491cd73a3b92580fdef0b5acae67fef594b1: Status 404 returned error can't find the container with id e48c7e2fc75cc5592f34c3e20818491cd73a3b92580fdef0b5acae67fef594b1
Sep 17 17:41:37.066058 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:37.066026    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:37.066262 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:37.066099    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f80b68ad-44e6-40fd-828b-aa73ab059235-metrics-tls\") pod \"dns-default-qjlh9\" (UID: \"f80b68ad-44e6-40fd-828b-aa73ab059235\") " pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:37.066262 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:37.066195    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:37.066262 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:37.066252    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f80b68ad-44e6-40fd-828b-aa73ab059235-metrics-tls podName:f80b68ad-44e6-40fd-828b-aa73ab059235 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:38.066237345 +0000 UTC m=+37.659074305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f80b68ad-44e6-40fd-828b-aa73ab059235-metrics-tls") pod "dns-default-qjlh9" (UID: "f80b68ad-44e6-40fd-828b-aa73ab059235") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:37.066422 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:37.066200    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:41:37.066422 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:37.066297    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67458c6fcd-jkctp: secret "image-registry-tls" not found
Sep 17 17:41:37.066422 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:37.066360    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls podName:724b2fc7-ecc7-4b88-98ff-79188ad6d792 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:38.066340615 +0000 UTC m=+37.659177597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls") pod "image-registry-67458c6fcd-jkctp" (UID: "724b2fc7-ecc7-4b88-98ff-79188ad6d792") : secret "image-registry-tls" not found
Sep 17 17:41:37.532196 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:37.532158    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:41:37.532196 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:37.532174    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:41:37.536644 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:37.536611    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:41:37.537197 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:37.537103    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:41:37.537715 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:37.537672    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gwk7t\""
Sep 17 17:41:37.714307 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:37.714259    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5zhfj" event={"ID":"2353bb3f-e217-4fb7-9202-8c567d93cf29","Type":"ContainerStarted","Data":"e48c7e2fc75cc5592f34c3e20818491cd73a3b92580fdef0b5acae67fef594b1"}
Sep 17 17:41:37.718099 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:37.718073    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7qls8" event={"ID":"0623a097-d7b1-4be4-be54-a29b81cc540a","Type":"ContainerStarted","Data":"e2974949b0fbd73d06cc8259543426ac4bb3844c77e45f9f32fd465c2a881c14"}
Sep 17 17:41:37.740457 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:37.740402    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7qls8" podStartSLOduration=9.799091171 podStartE2EDuration="35.740382399s" totalImagesPullingTime="25.941291228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:37.739387174 +0000 UTC m=+37.332224169" watchObservedRunningTime="2026-09-17 17:41:37.740382399 +0000 UTC m=+37.333219381"
Sep 17 17:41:38.073088 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:38.073059    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f80b68ad-44e6-40fd-828b-aa73ab059235-metrics-tls\") pod \"dns-default-qjlh9\" (UID: \"f80b68ad-44e6-40fd-828b-aa73ab059235\") " pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:38.073247 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:38.073101    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:38.073247 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:38.073234    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:38.073358 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:38.073313    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f80b68ad-44e6-40fd-828b-aa73ab059235-metrics-tls podName:f80b68ad-44e6-40fd-828b-aa73ab059235 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:40.07329029 +0000 UTC m=+39.666127265 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f80b68ad-44e6-40fd-828b-aa73ab059235-metrics-tls") pod "dns-default-qjlh9" (UID: "f80b68ad-44e6-40fd-828b-aa73ab059235") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:38.073358 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:38.073313    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:41:38.073358 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:38.073336    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67458c6fcd-jkctp: secret "image-registry-tls" not found
Sep 17 17:41:38.073474 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:38.073394    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls podName:724b2fc7-ecc7-4b88-98ff-79188ad6d792 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:40.073377697 +0000 UTC m=+39.666214676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls") pod "image-registry-67458c6fcd-jkctp" (UID: "724b2fc7-ecc7-4b88-98ff-79188ad6d792") : secret "image-registry-tls" not found
Sep 17 17:41:39.724251 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:39.724088    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5zhfj" event={"ID":"2353bb3f-e217-4fb7-9202-8c567d93cf29","Type":"ContainerStarted","Data":"3a705578bf7bc2052cde057c0d648e98315a8539ef1a6df040dd59efd9596612"}
Sep 17 17:41:39.740908 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:39.740868    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5zhfj" podStartSLOduration=1.4587589 podStartE2EDuration="3.740853206s" totalImagesPullingTime="2.282094306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:39.73961977 +0000 UTC m=+39.332456750" watchObservedRunningTime="2026-09-17 17:41:39.740853206 +0000 UTC m=+39.333690234"
Sep 17 17:41:40.090479 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:40.090448    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:40.090663 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:40.090514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f80b68ad-44e6-40fd-828b-aa73ab059235-metrics-tls\") pod \"dns-default-qjlh9\" (UID: \"f80b68ad-44e6-40fd-828b-aa73ab059235\") " pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:40.090663 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:40.090609    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:40.090663 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:40.090620    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:41:40.090663 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:40.090641    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67458c6fcd-jkctp: secret "image-registry-tls" not found
Sep 17 17:41:40.090663 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:40.090660    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f80b68ad-44e6-40fd-828b-aa73ab059235-metrics-tls podName:f80b68ad-44e6-40fd-828b-aa73ab059235 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:44.090647117 +0000 UTC m=+43.683484078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f80b68ad-44e6-40fd-828b-aa73ab059235-metrics-tls") pod "dns-default-qjlh9" (UID: "f80b68ad-44e6-40fd-828b-aa73ab059235") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:40.090857 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:40.090706    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls podName:724b2fc7-ecc7-4b88-98ff-79188ad6d792 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:44.09068832 +0000 UTC m=+43.683525281 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls") pod "image-registry-67458c6fcd-jkctp" (UID: "724b2fc7-ecc7-4b88-98ff-79188ad6d792") : secret "image-registry-tls" not found
Sep 17 17:41:42.164170 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.164138    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xrbg5"]
Sep 17 17:41:42.192411 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.192385    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xrbg5"]
Sep 17 17:41:42.192537 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.192487    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:42.195160 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.195139    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:41:42.195367 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.195139    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:41:42.195453 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.195150    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:41:42.195453 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.195158    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cmvbs\""
Sep 17 17:41:42.195547 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.195484    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:41:42.305770 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.305737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c6fd988-98c7-4e47-b583-64625bdb7cb2-crio-socket\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:42.305936 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.305782    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c6fd988-98c7-4e47-b583-64625bdb7cb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:42.305936 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.305833    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c6fd988-98c7-4e47-b583-64625bdb7cb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:42.305936 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.305852    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c6fd988-98c7-4e47-b583-64625bdb7cb2-data-volume\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:42.305936 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.305868    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5268f\" (UniqueName: \"kubernetes.io/projected/1c6fd988-98c7-4e47-b583-64625bdb7cb2-kube-api-access-5268f\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:42.406095 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.406068    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5268f\" (UniqueName: \"kubernetes.io/projected/1c6fd988-98c7-4e47-b583-64625bdb7cb2-kube-api-access-5268f\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:42.406236 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.406118    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c6fd988-98c7-4e47-b583-64625bdb7cb2-crio-socket\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:42.406236 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.406148    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c6fd988-98c7-4e47-b583-64625bdb7cb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:42.406236 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.406195    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c6fd988-98c7-4e47-b583-64625bdb7cb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:42.406236 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.406211    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c6fd988-98c7-4e47-b583-64625bdb7cb2-data-volume\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:42.406397 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:42.406295    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:41:42.406397 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:42.406313    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:42.406397 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:42.406366    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c6fd988-98c7-4e47-b583-64625bdb7cb2-kube-rbac-proxy-cm podName:1c6fd988-98c7-4e47-b583-64625bdb7cb2 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:42.906347586 +0000 UTC m=+42.499184558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c6fd988-98c7-4e47-b583-64625bdb7cb2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xrbg5" (UID: "1c6fd988-98c7-4e47-b583-64625bdb7cb2") : configmap "kube-rbac-proxy" not found
Sep 17 17:41:42.406397 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:42.406387    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6fd988-98c7-4e47-b583-64625bdb7cb2-insights-runtime-extractor-tls podName:1c6fd988-98c7-4e47-b583-64625bdb7cb2 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:42.906377579 +0000 UTC m=+42.499214538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c6fd988-98c7-4e47-b583-64625bdb7cb2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xrbg5" (UID: "1c6fd988-98c7-4e47-b583-64625bdb7cb2") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:42.406531 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.406408    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c6fd988-98c7-4e47-b583-64625bdb7cb2-crio-socket\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:42.406531 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.406445    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c6fd988-98c7-4e47-b583-64625bdb7cb2-data-volume\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:42.422252 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.422200    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5268f\" (UniqueName: \"kubernetes.io/projected/1c6fd988-98c7-4e47-b583-64625bdb7cb2-kube-api-access-5268f\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:42.910144 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.910102    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c6fd988-98c7-4e47-b583-64625bdb7cb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:42.910318 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:42.910178    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c6fd988-98c7-4e47-b583-64625bdb7cb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:42.910318 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:42.910262    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:41:42.910318 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:42.910266    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:42.910456 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:42.910326    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c6fd988-98c7-4e47-b583-64625bdb7cb2-kube-rbac-proxy-cm podName:1c6fd988-98c7-4e47-b583-64625bdb7cb2 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:43.910310374 +0000 UTC m=+43.503147333 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c6fd988-98c7-4e47-b583-64625bdb7cb2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xrbg5" (UID: "1c6fd988-98c7-4e47-b583-64625bdb7cb2") : configmap "kube-rbac-proxy" not found
Sep 17 17:41:42.910456 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:42.910340    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6fd988-98c7-4e47-b583-64625bdb7cb2-insights-runtime-extractor-tls podName:1c6fd988-98c7-4e47-b583-64625bdb7cb2 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:43.910334431 +0000 UTC m=+43.503171404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c6fd988-98c7-4e47-b583-64625bdb7cb2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xrbg5" (UID: "1c6fd988-98c7-4e47-b583-64625bdb7cb2") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:43.916836 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:43.916798    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c6fd988-98c7-4e47-b583-64625bdb7cb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:43.917246 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:43.916869    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c6fd988-98c7-4e47-b583-64625bdb7cb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:43.917246 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:43.916948    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:41:43.917246 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:43.916996    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c6fd988-98c7-4e47-b583-64625bdb7cb2-kube-rbac-proxy-cm podName:1c6fd988-98c7-4e47-b583-64625bdb7cb2 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:45.916982907 +0000 UTC m=+45.509819867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c6fd988-98c7-4e47-b583-64625bdb7cb2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xrbg5" (UID: "1c6fd988-98c7-4e47-b583-64625bdb7cb2") : configmap "kube-rbac-proxy" not found
Sep 17 17:41:43.917246 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:43.916944    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:43.917246 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:43.917085    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6fd988-98c7-4e47-b583-64625bdb7cb2-insights-runtime-extractor-tls podName:1c6fd988-98c7-4e47-b583-64625bdb7cb2 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:45.91707069 +0000 UTC m=+45.509907651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c6fd988-98c7-4e47-b583-64625bdb7cb2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xrbg5" (UID: "1c6fd988-98c7-4e47-b583-64625bdb7cb2") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:44.117981 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:44.117949    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f80b68ad-44e6-40fd-828b-aa73ab059235-metrics-tls\") pod \"dns-default-qjlh9\" (UID: \"f80b68ad-44e6-40fd-828b-aa73ab059235\") " pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:44.118144 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:44.118000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:44.118144 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:44.118097    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:41:44.118144 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:44.118108    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67458c6fcd-jkctp: secret "image-registry-tls" not found
Sep 17 17:41:44.118144 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:44.118109    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:41:44.118288 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:44.118154    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls podName:724b2fc7-ecc7-4b88-98ff-79188ad6d792 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:52.118142011 +0000 UTC m=+51.710978972 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls") pod "image-registry-67458c6fcd-jkctp" (UID: "724b2fc7-ecc7-4b88-98ff-79188ad6d792") : secret "image-registry-tls" not found
Sep 17 17:41:44.118288 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:44.118166    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f80b68ad-44e6-40fd-828b-aa73ab059235-metrics-tls podName:f80b68ad-44e6-40fd-828b-aa73ab059235 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:52.118160363 +0000 UTC m=+51.710997324 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f80b68ad-44e6-40fd-828b-aa73ab059235-metrics-tls") pod "dns-default-qjlh9" (UID: "f80b68ad-44e6-40fd-828b-aa73ab059235") : secret "dns-default-metrics-tls" not found
Sep 17 17:41:44.332075 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:44.332052    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vmxsb_d54d9137-d225-4657-b601-b1a8bdee3b68/dns-node-resolver/0.log"
Sep 17 17:41:44.903523 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:44.903490    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-jr9lk"]
Sep 17 17:41:44.918174 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:44.918148    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-jr9lk"]
Sep 17 17:41:44.918537 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:44.918245    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-jr9lk"
Sep 17 17:41:44.921036 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:44.921014    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 17:41:44.921036 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:44.921032    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 17:41:44.921196 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:44.921110    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 17:41:44.922070 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:44.922050    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 17:41:44.922245 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:44.922230    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bn2bt\""
Sep 17 17:41:44.930799 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:44.930782    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7rmhq_0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852/node-ca/0.log"
Sep 17 17:41:45.026363 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:45.026330    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs2zr\" (UniqueName: \"kubernetes.io/projected/641b8201-927c-43c0-aaf4-66c3e9f6b3cc-kube-api-access-cs2zr\") pod \"service-ca-55f6fc7b86-jr9lk\" (UID: \"641b8201-927c-43c0-aaf4-66c3e9f6b3cc\") " pod="openshift-service-ca/service-ca-55f6fc7b86-jr9lk"
Sep 17 17:41:45.026527 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:45.026397    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/641b8201-927c-43c0-aaf4-66c3e9f6b3cc-signing-cabundle\") pod \"service-ca-55f6fc7b86-jr9lk\" (UID: \"641b8201-927c-43c0-aaf4-66c3e9f6b3cc\") " pod="openshift-service-ca/service-ca-55f6fc7b86-jr9lk"
Sep 17 17:41:45.026527 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:45.026441    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/641b8201-927c-43c0-aaf4-66c3e9f6b3cc-signing-key\") pod \"service-ca-55f6fc7b86-jr9lk\" (UID: \"641b8201-927c-43c0-aaf4-66c3e9f6b3cc\") " pod="openshift-service-ca/service-ca-55f6fc7b86-jr9lk"
Sep 17 17:41:45.127139 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:45.127100    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/641b8201-927c-43c0-aaf4-66c3e9f6b3cc-signing-cabundle\") pod \"service-ca-55f6fc7b86-jr9lk\" (UID: \"641b8201-927c-43c0-aaf4-66c3e9f6b3cc\") " pod="openshift-service-ca/service-ca-55f6fc7b86-jr9lk"
Sep 17 17:41:45.127320 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:45.127142    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/641b8201-927c-43c0-aaf4-66c3e9f6b3cc-signing-key\") pod \"service-ca-55f6fc7b86-jr9lk\" (UID: \"641b8201-927c-43c0-aaf4-66c3e9f6b3cc\") " pod="openshift-service-ca/service-ca-55f6fc7b86-jr9lk"
Sep 17 17:41:45.127381 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:45.127343    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs2zr\" (UniqueName: \"kubernetes.io/projected/641b8201-927c-43c0-aaf4-66c3e9f6b3cc-kube-api-access-cs2zr\") pod \"service-ca-55f6fc7b86-jr9lk\" (UID: \"641b8201-927c-43c0-aaf4-66c3e9f6b3cc\") " pod="openshift-service-ca/service-ca-55f6fc7b86-jr9lk"
Sep 17 17:41:45.127841 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:45.127813    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/641b8201-927c-43c0-aaf4-66c3e9f6b3cc-signing-cabundle\") pod \"service-ca-55f6fc7b86-jr9lk\" (UID: \"641b8201-927c-43c0-aaf4-66c3e9f6b3cc\") " pod="openshift-service-ca/service-ca-55f6fc7b86-jr9lk"
Sep 17 17:41:45.129532 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:45.129510    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/641b8201-927c-43c0-aaf4-66c3e9f6b3cc-signing-key\") pod \"service-ca-55f6fc7b86-jr9lk\" (UID: \"641b8201-927c-43c0-aaf4-66c3e9f6b3cc\") " pod="openshift-service-ca/service-ca-55f6fc7b86-jr9lk"
Sep 17 17:41:45.135158 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:45.135135    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs2zr\" (UniqueName: \"kubernetes.io/projected/641b8201-927c-43c0-aaf4-66c3e9f6b3cc-kube-api-access-cs2zr\") pod \"service-ca-55f6fc7b86-jr9lk\" (UID: \"641b8201-927c-43c0-aaf4-66c3e9f6b3cc\") " pod="openshift-service-ca/service-ca-55f6fc7b86-jr9lk"
Sep 17 17:41:45.227289 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:45.227206    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-jr9lk"
Sep 17 17:41:45.523607 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:45.523520    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-jr9lk"]
Sep 17 17:41:45.527964 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:45.527937    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641b8201_927c_43c0_aaf4_66c3e9f6b3cc.slice/crio-138466f1b151a596922d9a31855455072126c4f2bdf32c0f69aeb8cf5cdcf837 WatchSource:0}: Error finding container 138466f1b151a596922d9a31855455072126c4f2bdf32c0f69aeb8cf5cdcf837: Status 404 returned error can't find the container with id 138466f1b151a596922d9a31855455072126c4f2bdf32c0f69aeb8cf5cdcf837
Sep 17 17:41:45.731664 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:45.731634    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5zhfj_2353bb3f-e217-4fb7-9202-8c567d93cf29/serve-healthcheck-canary/0.log"
Sep 17 17:41:45.738215 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:45.738190    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-jr9lk" event={"ID":"641b8201-927c-43c0-aaf4-66c3e9f6b3cc","Type":"ContainerStarted","Data":"138466f1b151a596922d9a31855455072126c4f2bdf32c0f69aeb8cf5cdcf837"}
Sep 17 17:41:45.932952 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:45.932920    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c6fd988-98c7-4e47-b583-64625bdb7cb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:45.933326 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:45.932981    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c6fd988-98c7-4e47-b583-64625bdb7cb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:45.933326 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:45.933047    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:41:45.933326 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:45.933077    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:45.933326 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:45.933125    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c6fd988-98c7-4e47-b583-64625bdb7cb2-kube-rbac-proxy-cm podName:1c6fd988-98c7-4e47-b583-64625bdb7cb2 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:49.933108761 +0000 UTC m=+49.525945725 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c6fd988-98c7-4e47-b583-64625bdb7cb2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xrbg5" (UID: "1c6fd988-98c7-4e47-b583-64625bdb7cb2") : configmap "kube-rbac-proxy" not found
Sep 17 17:41:45.933326 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:45.933139    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6fd988-98c7-4e47-b583-64625bdb7cb2-insights-runtime-extractor-tls podName:1c6fd988-98c7-4e47-b583-64625bdb7cb2 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:49.933133126 +0000 UTC m=+49.525970085 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c6fd988-98c7-4e47-b583-64625bdb7cb2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xrbg5" (UID: "1c6fd988-98c7-4e47-b583-64625bdb7cb2") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:48.747264 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:48.747225    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-jr9lk" event={"ID":"641b8201-927c-43c0-aaf4-66c3e9f6b3cc","Type":"ContainerStarted","Data":"ca6794da7d8bec371b3c3c8b1404b8e5e7602a76afa579ace8f887ec612842eb"}
Sep 17 17:41:48.766580 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:48.766521    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-jr9lk" podStartSLOduration=2.512297556 podStartE2EDuration="4.766506883s" totalImagesPullingTime="2.254209327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:48.764932657 +0000 UTC m=+48.357769638" watchObservedRunningTime="2026-09-17 17:41:48.766506883 +0000 UTC m=+48.359343865"
Sep 17 17:41:49.962778 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:49.962743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c6fd988-98c7-4e47-b583-64625bdb7cb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:49.963119 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:49.962809    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c6fd988-98c7-4e47-b583-64625bdb7cb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:49.963119 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:49.962869    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:49.963119 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:49.962926    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6fd988-98c7-4e47-b583-64625bdb7cb2-insights-runtime-extractor-tls podName:1c6fd988-98c7-4e47-b583-64625bdb7cb2 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:57.962911235 +0000 UTC m=+57.555748195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c6fd988-98c7-4e47-b583-64625bdb7cb2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xrbg5" (UID: "1c6fd988-98c7-4e47-b583-64625bdb7cb2") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:41:49.963256 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:49.963240    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c6fd988-98c7-4e47-b583-64625bdb7cb2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:52.178825 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:52.178791    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f80b68ad-44e6-40fd-828b-aa73ab059235-metrics-tls\") pod \"dns-default-qjlh9\" (UID: \"f80b68ad-44e6-40fd-828b-aa73ab059235\") " pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:52.179239 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:52.178852    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:41:52.179239 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:52.178963    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:41:52.179239 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:52.178976    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67458c6fcd-jkctp: secret "image-registry-tls" not found
Sep 17 17:41:52.179239 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:41:52.179033    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls podName:724b2fc7-ecc7-4b88-98ff-79188ad6d792 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:08.179016547 +0000 UTC m=+67.771853514 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls") pod "image-registry-67458c6fcd-jkctp" (UID: "724b2fc7-ecc7-4b88-98ff-79188ad6d792") : secret "image-registry-tls" not found
Sep 17 17:41:52.181048 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:52.181026    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f80b68ad-44e6-40fd-828b-aa73ab059235-metrics-tls\") pod \"dns-default-qjlh9\" (UID: \"f80b68ad-44e6-40fd-828b-aa73ab059235\") " pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:52.260524 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:52.260495    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:52.565316 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:52.565249    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qjlh9"]
Sep 17 17:41:52.568712 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:52.568687    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80b68ad_44e6_40fd_828b_aa73ab059235.slice/crio-5e998c65071f57538f2d1302b59a636198a0ad1e17f87d642bf512e363954867 WatchSource:0}: Error finding container 5e998c65071f57538f2d1302b59a636198a0ad1e17f87d642bf512e363954867: Status 404 returned error can't find the container with id 5e998c65071f57538f2d1302b59a636198a0ad1e17f87d642bf512e363954867
Sep 17 17:41:52.756939 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:52.756909    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qjlh9" event={"ID":"f80b68ad-44e6-40fd-828b-aa73ab059235","Type":"ContainerStarted","Data":"5e998c65071f57538f2d1302b59a636198a0ad1e17f87d642bf512e363954867"}
Sep 17 17:41:54.764837 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:54.764806    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qjlh9" event={"ID":"f80b68ad-44e6-40fd-828b-aa73ab059235","Type":"ContainerStarted","Data":"e71d7cabaa59842de9bc1260353c902e6d0d8142d42d938c12b7d4992932e0d4"}
Sep 17 17:41:54.765124 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:54.764868    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qjlh9" event={"ID":"f80b68ad-44e6-40fd-828b-aa73ab059235","Type":"ContainerStarted","Data":"388dbce6c6182857c97818b18e3864e22ea1f73b4d2ef93b1186109edfbeba61"}
Sep 17 17:41:55.767311 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:55.767281    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:41:55.793574 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:55.793522    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qjlh9" podStartSLOduration=17.879909107 podStartE2EDuration="19.79350944s" totalImagesPullingTime="1.913600333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:55.793293207 +0000 UTC m=+55.386130190" watchObservedRunningTime="2026-09-17 17:41:55.79350944 +0000 UTC m=+55.386346421"
Sep 17 17:41:58.024481 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:58.024441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c6fd988-98c7-4e47-b583-64625bdb7cb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:58.026517 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:58.026498    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c6fd988-98c7-4e47-b583-64625bdb7cb2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xrbg5\" (UID: \"1c6fd988-98c7-4e47-b583-64625bdb7cb2\") " pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:58.101035 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:58.101008    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xrbg5"
Sep 17 17:41:58.377922 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:58.377888    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xrbg5"]
Sep 17 17:41:58.381374 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:41:58.381349    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6fd988_98c7_4e47_b583_64625bdb7cb2.slice/crio-9f0d534d656da4a167a9c712e59772fb6e5e93d5fb0eb906f9c1bb7304f464dc WatchSource:0}: Error finding container 9f0d534d656da4a167a9c712e59772fb6e5e93d5fb0eb906f9c1bb7304f464dc: Status 404 returned error can't find the container with id 9f0d534d656da4a167a9c712e59772fb6e5e93d5fb0eb906f9c1bb7304f464dc
Sep 17 17:41:58.776084 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:58.776015    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xrbg5" event={"ID":"1c6fd988-98c7-4e47-b583-64625bdb7cb2","Type":"ContainerStarted","Data":"c9e663a2d0c002c89500c7d10a284cb253b171d2b77b7b906cdb732fd4856b26"}
Sep 17 17:41:58.776084 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:58.776064    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xrbg5" event={"ID":"1c6fd988-98c7-4e47-b583-64625bdb7cb2","Type":"ContainerStarted","Data":"9f0d534d656da4a167a9c712e59772fb6e5e93d5fb0eb906f9c1bb7304f464dc"}
Sep 17 17:41:59.781250 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:41:59.781213    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xrbg5" event={"ID":"1c6fd988-98c7-4e47-b583-64625bdb7cb2","Type":"ContainerStarted","Data":"1fc7ea4360d9a2bd3a9f7fb4d89d2fd9b4730b59faf0024be77420ffb6286bcb"}
Sep 17 17:42:00.785808 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:00.785777    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xrbg5" event={"ID":"1c6fd988-98c7-4e47-b583-64625bdb7cb2","Type":"ContainerStarted","Data":"fa54c6e60883bf47716af7a18aac4c4ed186613a71c655d32cd85a2d5b87ef28"}
Sep 17 17:42:01.700107 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:01.700082    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s4nxf"
Sep 17 17:42:01.732920 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:01.732871    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xrbg5" podStartSLOduration=18.013806272 podStartE2EDuration="19.73285529s" totalImagesPullingTime="1.719049018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:00.810977328 +0000 UTC m=+60.403814309" watchObservedRunningTime="2026-09-17 17:42:01.73285529 +0000 UTC m=+61.325692272"
Sep 17 17:42:05.771876 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:05.771849    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qjlh9"
Sep 17 17:42:07.024526 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.024468    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67458c6fcd-jkctp"]
Sep 17 17:42:07.024885 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:42:07.024686    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-67458c6fcd-jkctp" podUID="724b2fc7-ecc7-4b88-98ff-79188ad6d792"
Sep 17 17:42:07.033648 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.033626    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"]
Sep 17 17:42:07.039653 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.039638    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.044954 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.044925    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 17:42:07.045063 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.044965    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 17:42:07.045063 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.044977    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 17:42:07.045948 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.045932    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 17:42:07.046143 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.046131    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 17:42:07.046204 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.046170    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 17:42:07.047882 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.047866    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 17:42:07.059772 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.059751    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"]
Sep 17 17:42:07.184818 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.184793    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9abf0833-ff08-4673-b539-d1cbfe62edaf-hub\") pod \"cluster-proxy-proxy-agent-64df54d8d4-6frh8\" (UID: \"9abf0833-ff08-4673-b539-d1cbfe62edaf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.184913 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.184824    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs77f\" (UniqueName: \"kubernetes.io/projected/9abf0833-ff08-4673-b539-d1cbfe62edaf-kube-api-access-qs77f\") pod \"cluster-proxy-proxy-agent-64df54d8d4-6frh8\" (UID: \"9abf0833-ff08-4673-b539-d1cbfe62edaf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.184963 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.184940    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9abf0833-ff08-4673-b539-d1cbfe62edaf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64df54d8d4-6frh8\" (UID: \"9abf0833-ff08-4673-b539-d1cbfe62edaf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.185012 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.184967    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9abf0833-ff08-4673-b539-d1cbfe62edaf-ca\") pod \"cluster-proxy-proxy-agent-64df54d8d4-6frh8\" (UID: \"9abf0833-ff08-4673-b539-d1cbfe62edaf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.185012 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.184985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9abf0833-ff08-4673-b539-d1cbfe62edaf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64df54d8d4-6frh8\" (UID: \"9abf0833-ff08-4673-b539-d1cbfe62edaf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.185012 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.185000    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9abf0833-ff08-4673-b539-d1cbfe62edaf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64df54d8d4-6frh8\" (UID: \"9abf0833-ff08-4673-b539-d1cbfe62edaf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.285492 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.285443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9abf0833-ff08-4673-b539-d1cbfe62edaf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64df54d8d4-6frh8\" (UID: \"9abf0833-ff08-4673-b539-d1cbfe62edaf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.285492 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.285472    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9abf0833-ff08-4673-b539-d1cbfe62edaf-ca\") pod \"cluster-proxy-proxy-agent-64df54d8d4-6frh8\" (UID: \"9abf0833-ff08-4673-b539-d1cbfe62edaf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.285492 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.285487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9abf0833-ff08-4673-b539-d1cbfe62edaf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64df54d8d4-6frh8\" (UID: \"9abf0833-ff08-4673-b539-d1cbfe62edaf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.285718 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.285502    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9abf0833-ff08-4673-b539-d1cbfe62edaf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64df54d8d4-6frh8\" (UID: \"9abf0833-ff08-4673-b539-d1cbfe62edaf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.285718 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.285525    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9abf0833-ff08-4673-b539-d1cbfe62edaf-hub\") pod \"cluster-proxy-proxy-agent-64df54d8d4-6frh8\" (UID: \"9abf0833-ff08-4673-b539-d1cbfe62edaf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.285718 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.285539    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs77f\" (UniqueName: \"kubernetes.io/projected/9abf0833-ff08-4673-b539-d1cbfe62edaf-kube-api-access-qs77f\") pod \"cluster-proxy-proxy-agent-64df54d8d4-6frh8\" (UID: \"9abf0833-ff08-4673-b539-d1cbfe62edaf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.286299 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.286276    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9abf0833-ff08-4673-b539-d1cbfe62edaf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64df54d8d4-6frh8\" (UID: \"9abf0833-ff08-4673-b539-d1cbfe62edaf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.288069 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.288044    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9abf0833-ff08-4673-b539-d1cbfe62edaf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64df54d8d4-6frh8\" (UID: \"9abf0833-ff08-4673-b539-d1cbfe62edaf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.288069 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.288057    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9abf0833-ff08-4673-b539-d1cbfe62edaf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64df54d8d4-6frh8\" (UID: \"9abf0833-ff08-4673-b539-d1cbfe62edaf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.288344 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.288328    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9abf0833-ff08-4673-b539-d1cbfe62edaf-ca\") pod \"cluster-proxy-proxy-agent-64df54d8d4-6frh8\" (UID: \"9abf0833-ff08-4673-b539-d1cbfe62edaf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.288463 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.288444    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9abf0833-ff08-4673-b539-d1cbfe62edaf-hub\") pod \"cluster-proxy-proxy-agent-64df54d8d4-6frh8\" (UID: \"9abf0833-ff08-4673-b539-d1cbfe62edaf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.311343 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.311321    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs77f\" (UniqueName: \"kubernetes.io/projected/9abf0833-ff08-4673-b539-d1cbfe62edaf-kube-api-access-qs77f\") pod \"cluster-proxy-proxy-agent-64df54d8d4-6frh8\" (UID: \"9abf0833-ff08-4673-b539-d1cbfe62edaf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.363362 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.363337    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"
Sep 17 17:42:07.685812 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.685783    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8"]
Sep 17 17:42:07.689515 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:42:07.689492    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9abf0833_ff08_4673_b539_d1cbfe62edaf.slice/crio-cbc830fdebec5df3d7a9e3e700f4cf88412e52aa0198391d195ae648d96ae00a WatchSource:0}: Error finding container cbc830fdebec5df3d7a9e3e700f4cf88412e52aa0198391d195ae648d96ae00a: Status 404 returned error can't find the container with id cbc830fdebec5df3d7a9e3e700f4cf88412e52aa0198391d195ae648d96ae00a
Sep 17 17:42:07.806852 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.806828    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:42:07.806852 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.806842    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8" event={"ID":"9abf0833-ff08-4673-b539-d1cbfe62edaf","Type":"ContainerStarted","Data":"cbc830fdebec5df3d7a9e3e700f4cf88412e52aa0198391d195ae648d96ae00a"}
Sep 17 17:42:07.839039 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.839016    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:42:07.990841 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.990781    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mt9r5\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-kube-api-access-mt9r5\") pod \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") "
Sep 17 17:42:07.990841 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.990823    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/724b2fc7-ecc7-4b88-98ff-79188ad6d792-ca-trust-extracted\") pod \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") "
Sep 17 17:42:07.990841 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.990840    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/724b2fc7-ecc7-4b88-98ff-79188ad6d792-trusted-ca\") pod \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") "
Sep 17 17:42:07.991086 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.990866    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/724b2fc7-ecc7-4b88-98ff-79188ad6d792-installation-pull-secrets\") pod \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") "
Sep 17 17:42:07.991086 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.990886    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-bound-sa-token\") pod \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") "
Sep 17 17:42:07.991086 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.990914    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/724b2fc7-ecc7-4b88-98ff-79188ad6d792-image-registry-private-configuration\") pod \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") "
Sep 17 17:42:07.991086 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.990944    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-certificates\") pod \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") "
Sep 17 17:42:07.991292 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.991123    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724b2fc7-ecc7-4b88-98ff-79188ad6d792-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "724b2fc7-ecc7-4b88-98ff-79188ad6d792" (UID: "724b2fc7-ecc7-4b88-98ff-79188ad6d792"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:42:07.991344 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.991311    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724b2fc7-ecc7-4b88-98ff-79188ad6d792-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "724b2fc7-ecc7-4b88-98ff-79188ad6d792" (UID: "724b2fc7-ecc7-4b88-98ff-79188ad6d792"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:07.991420 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.991394    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "724b2fc7-ecc7-4b88-98ff-79188ad6d792" (UID: "724b2fc7-ecc7-4b88-98ff-79188ad6d792"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:42:07.993043 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.993009    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724b2fc7-ecc7-4b88-98ff-79188ad6d792-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "724b2fc7-ecc7-4b88-98ff-79188ad6d792" (UID: "724b2fc7-ecc7-4b88-98ff-79188ad6d792"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:42:07.993043 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.993034    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-kube-api-access-mt9r5" (OuterVolumeSpecName: "kube-api-access-mt9r5") pod "724b2fc7-ecc7-4b88-98ff-79188ad6d792" (UID: "724b2fc7-ecc7-4b88-98ff-79188ad6d792"). InnerVolumeSpecName "kube-api-access-mt9r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:07.993159 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.993073    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "724b2fc7-ecc7-4b88-98ff-79188ad6d792" (UID: "724b2fc7-ecc7-4b88-98ff-79188ad6d792"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:07.993159 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:07.993073    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724b2fc7-ecc7-4b88-98ff-79188ad6d792-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "724b2fc7-ecc7-4b88-98ff-79188ad6d792" (UID: "724b2fc7-ecc7-4b88-98ff-79188ad6d792"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:42:08.092072 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.092050    2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/724b2fc7-ecc7-4b88-98ff-79188ad6d792-ca-trust-extracted\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:42:08.092072 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.092068    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/724b2fc7-ecc7-4b88-98ff-79188ad6d792-trusted-ca\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:42:08.092388 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.092077    2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/724b2fc7-ecc7-4b88-98ff-79188ad6d792-installation-pull-secrets\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:42:08.092388 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.092087    2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-bound-sa-token\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:42:08.092388 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.092095    2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/724b2fc7-ecc7-4b88-98ff-79188ad6d792-image-registry-private-configuration\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:42:08.092388 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.092104    2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-certificates\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:42:08.092388 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.092113    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt9r5\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-kube-api-access-mt9r5\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:42:08.192460 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.192429    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret\") pod \"global-pull-secret-syncer-7cdxt\" (UID: \"a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2\") " pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:42:08.192633 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.192478    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:42:08.192633 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.192524    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll9v7\" (UniqueName: \"kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7\") pod \"network-check-target-qrrjr\" (UID: \"365faef8-3867-42a8-a227-d81293771f14\") " pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:42:08.195018 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.194994    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls\") pod \"image-registry-67458c6fcd-jkctp\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") " pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:42:08.196104 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.196083    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:42:08.196208 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.196119    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:42:08.205638 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.205618    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2-original-pull-secret\") pod \"global-pull-secret-syncer-7cdxt\" (UID: \"a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2\") " pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:42:08.205958 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.205929    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:42:08.216814 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.216786    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll9v7\" (UniqueName: \"kubernetes.io/projected/365faef8-3867-42a8-a227-d81293771f14-kube-api-access-ll9v7\") pod \"network-check-target-qrrjr\" (UID: \"365faef8-3867-42a8-a227-d81293771f14\") " pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:42:08.293791 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.293713    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs\") pod \"network-metrics-daemon-z44z2\" (UID: \"84810a50-2551-4d78-ac6d-4d17f1524389\") " pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:42:08.296755 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.296719    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:42:08.306213 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.306192    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84810a50-2551-4d78-ac6d-4d17f1524389-metrics-certs\") pod \"network-metrics-daemon-z44z2\" (UID: \"84810a50-2551-4d78-ac6d-4d17f1524389\") " pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:42:08.343456 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.343432    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hq5mj\""
Sep 17 17:42:08.350795 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.350769    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:42:08.394025 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.393991    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls\") pod \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\" (UID: \"724b2fc7-ecc7-4b88-98ff-79188ad6d792\") "
Sep 17 17:42:08.397367 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.396936    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "724b2fc7-ecc7-4b88-98ff-79188ad6d792" (UID: "724b2fc7-ecc7-4b88-98ff-79188ad6d792"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:08.445133 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.445018    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7cdxt"
Sep 17 17:42:08.456179 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.456155    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gwk7t\""
Sep 17 17:42:08.461382 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.461361    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z44z2"
Sep 17 17:42:08.494949 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.494929    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/724b2fc7-ecc7-4b88-98ff-79188ad6d792-registry-tls\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:42:08.657840 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.657813    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qrrjr"]
Sep 17 17:42:08.661322 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:42:08.661297    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365faef8_3867_42a8_a227_d81293771f14.slice/crio-bcd648a8f975db2d259f77e9e2df6c12ab42e82680af095a5a57cbbde6b9fbe8 WatchSource:0}: Error finding container bcd648a8f975db2d259f77e9e2df6c12ab42e82680af095a5a57cbbde6b9fbe8: Status 404 returned error can't find the container with id bcd648a8f975db2d259f77e9e2df6c12ab42e82680af095a5a57cbbde6b9fbe8
Sep 17 17:42:08.743885 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.743857    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7cdxt"]
Sep 17 17:42:08.748046 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:42:08.748017    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda540f631_c6e3_4f4f_b9a5_4fbeb92f41e2.slice/crio-223cbe4590466ed719fb4763c7746295f19d2fe0c5851e153a56dc53ec01a5ad WatchSource:0}: Error finding container 223cbe4590466ed719fb4763c7746295f19d2fe0c5851e153a56dc53ec01a5ad: Status 404 returned error can't find the container with id 223cbe4590466ed719fb4763c7746295f19d2fe0c5851e153a56dc53ec01a5ad
Sep 17 17:42:08.775425 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.775396    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z44z2"]
Sep 17 17:42:08.778391 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:42:08.778366    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84810a50_2551_4d78_ac6d_4d17f1524389.slice/crio-c896637ade1551de327fe9b46c1ba506899d9d2544d2b83f4e51db637854a401 WatchSource:0}: Error finding container c896637ade1551de327fe9b46c1ba506899d9d2544d2b83f4e51db637854a401: Status 404 returned error can't find the container with id c896637ade1551de327fe9b46c1ba506899d9d2544d2b83f4e51db637854a401
Sep 17 17:42:08.812438 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.812378    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z44z2" event={"ID":"84810a50-2551-4d78-ac6d-4d17f1524389","Type":"ContainerStarted","Data":"c896637ade1551de327fe9b46c1ba506899d9d2544d2b83f4e51db637854a401"}
Sep 17 17:42:08.813703 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.813664    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7cdxt" event={"ID":"a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2","Type":"ContainerStarted","Data":"223cbe4590466ed719fb4763c7746295f19d2fe0c5851e153a56dc53ec01a5ad"}
Sep 17 17:42:08.814939 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.814921    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67458c6fcd-jkctp"
Sep 17 17:42:08.815028 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.814952    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qrrjr" event={"ID":"365faef8-3867-42a8-a227-d81293771f14","Type":"ContainerStarted","Data":"bcd648a8f975db2d259f77e9e2df6c12ab42e82680af095a5a57cbbde6b9fbe8"}
Sep 17 17:42:08.856104 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.856075    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67458c6fcd-jkctp"]
Sep 17 17:42:08.859809 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:08.859783    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67458c6fcd-jkctp"]
Sep 17 17:42:09.957284 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:09.957250    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zs2sz"]
Sep 17 17:42:09.970395 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:09.970370    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zs2sz"
Sep 17 17:42:09.974101 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:09.974072    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 17:42:09.974368 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:09.974346    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mrxck\""
Sep 17 17:42:09.994536 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:09.994511    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zs2sz"]
Sep 17 17:42:10.107293 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:10.107170    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fa773c89-0150-46a3-b187-08d5a995b39f-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-zs2sz\" (UID: \"fa773c89-0150-46a3-b187-08d5a995b39f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zs2sz"
Sep 17 17:42:10.207963 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:10.207818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fa773c89-0150-46a3-b187-08d5a995b39f-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-zs2sz\" (UID: \"fa773c89-0150-46a3-b187-08d5a995b39f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zs2sz"
Sep 17 17:42:10.211630 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:10.211578    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fa773c89-0150-46a3-b187-08d5a995b39f-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-zs2sz\" (UID: \"fa773c89-0150-46a3-b187-08d5a995b39f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zs2sz"
Sep 17 17:42:10.282579 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:10.282349    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zs2sz"
Sep 17 17:42:10.538043 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:10.537965    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="724b2fc7-ecc7-4b88-98ff-79188ad6d792" path="/var/lib/kubelet/pods/724b2fc7-ecc7-4b88-98ff-79188ad6d792/volumes"
Sep 17 17:42:10.576177 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:10.576105    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zs2sz"]
Sep 17 17:42:10.748065 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:42:10.748030    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa773c89_0150_46a3_b187_08d5a995b39f.slice/crio-efe1bd44f9fba9b014ca957d92a14e7d53e4ea9f4c7d7827ecd24a15b05730b4 WatchSource:0}: Error finding container efe1bd44f9fba9b014ca957d92a14e7d53e4ea9f4c7d7827ecd24a15b05730b4: Status 404 returned error can't find the container with id efe1bd44f9fba9b014ca957d92a14e7d53e4ea9f4c7d7827ecd24a15b05730b4
Sep 17 17:42:10.824509 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:10.824412    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zs2sz" event={"ID":"fa773c89-0150-46a3-b187-08d5a995b39f","Type":"ContainerStarted","Data":"efe1bd44f9fba9b014ca957d92a14e7d53e4ea9f4c7d7827ecd24a15b05730b4"}
Sep 17 17:42:15.843840 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:15.843805    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z44z2" event={"ID":"84810a50-2551-4d78-ac6d-4d17f1524389","Type":"ContainerStarted","Data":"52ad0a6abfe246b7e668ca04d6d74c5464ea3820dd1a8de5e03d05781ccac875"}
Sep 17 17:42:15.844290 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:15.843869    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z44z2" event={"ID":"84810a50-2551-4d78-ac6d-4d17f1524389","Type":"ContainerStarted","Data":"550ed4c787915080f4a8e0420c9717715959a0c1db5258b1eec58cd977a36350"}
Sep 17 17:42:15.845440 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:15.845412    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7cdxt" event={"ID":"a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2","Type":"ContainerStarted","Data":"9678f645758c0edea4b2a0d9a1cda6441de67bade1cc58417cd6febb5eef87bc"}
Sep 17 17:42:15.846832 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:15.846806    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qrrjr" event={"ID":"365faef8-3867-42a8-a227-d81293771f14","Type":"ContainerStarted","Data":"e8bce9d73889eb5e55d4e2e550528824c78be4d29228367038067c989c78b7b6"}
Sep 17 17:42:15.846951 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:15.846932    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:42:15.848798 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:15.848780    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8" event={"ID":"9abf0833-ff08-4673-b539-d1cbfe62edaf","Type":"ContainerStarted","Data":"3125d3660a5bd25e1c7c00a9662856dfb4716016afe6af8e1208dc00b2c75079"}
Sep 17 17:42:15.848908 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:15.848810    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8" event={"ID":"9abf0833-ff08-4673-b539-d1cbfe62edaf","Type":"ContainerStarted","Data":"328b5d0dad335d4074c722641603e24e02cbb25999d76723f74457cbdeecf346"}
Sep 17 17:42:15.848908 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:15.848828    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8" event={"ID":"9abf0833-ff08-4673-b539-d1cbfe62edaf","Type":"ContainerStarted","Data":"6f90324d49420aa45d430db618c6b5bb95f5967f41488ec9a1a6e67d3f3dd9a5"}
Sep 17 17:42:15.850205 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:15.850185    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zs2sz" event={"ID":"fa773c89-0150-46a3-b187-08d5a995b39f","Type":"ContainerStarted","Data":"866ed2b6a022726f2c8d84c88d5a274cbf6bc5a74227859239b1d209690a6730"}
Sep 17 17:42:15.850405 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:15.850391    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zs2sz"
Sep 17 17:42:15.854936 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:15.854920    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zs2sz"
Sep 17 17:42:15.883157 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:15.883115    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-z44z2" podStartSLOduration=68.145531385 podStartE2EDuration="1m13.88309856s" totalImagesPullingTime="5.737567175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:15.882613271 +0000 UTC m=+75.475450252" watchObservedRunningTime="2026-09-17 17:42:15.88309856 +0000 UTC m=+75.475935542"
Sep 17 17:42:15.950214 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:15.950175    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64df54d8d4-6frh8" podStartSLOduration=1.856722035 podStartE2EDuration="8.950161464s" totalImagesPullingTime="7.093439429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:15.919946178 +0000 UTC m=+75.512783159" watchObservedRunningTime="2026-09-17 17:42:15.950161464 +0000 UTC m=+75.542998445"
Sep 17 17:42:15.950317 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:15.950263    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qrrjr" podStartSLOduration=67.828820211 podStartE2EDuration="1m13.950259165s" totalImagesPullingTime="6.121438954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:15.949487976 +0000 UTC m=+75.542324960" watchObservedRunningTime="2026-09-17 17:42:15.950259165 +0000 UTC m=+75.543096147"
Sep 17 17:42:15.976979 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:15.976942    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7cdxt" podStartSLOduration=67.942031153 podStartE2EDuration="1m13.976934571s" totalImagesPullingTime="6.034903418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:15.975448871 +0000 UTC m=+75.568285851" watchObservedRunningTime="2026-09-17 17:42:15.976934571 +0000 UTC m=+75.569771552"
Sep 17 17:42:16.002324 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:16.002290    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-zs2sz" podStartSLOduration=2.935375041 podStartE2EDuration="7.002278787s" totalImagesPullingTime="4.066903746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:16.001675676 +0000 UTC m=+75.594512657" watchObservedRunningTime="2026-09-17 17:42:16.002278787 +0000 UTC m=+75.595115768"
Sep 17 17:42:21.554699 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.554660    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6"]
Sep 17 17:42:21.558631 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.558613    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6"
Sep 17 17:42:21.562856 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.562834    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 17:42:21.562971 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.562935    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:42:21.563888 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.563870    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:42:21.563966 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.563873    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:42:21.567098 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.567080    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:42:21.567285 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.567265    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-s6wrd\""
Sep 17 17:42:21.572243 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.572220    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6"]
Sep 17 17:42:21.587188 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.587164    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c61fd583-592e-4fee-b0fa-cc18e365b9ab-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-pdrc6\" (UID: \"c61fd583-592e-4fee-b0fa-cc18e365b9ab\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6"
Sep 17 17:42:21.587279 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.587205    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c61fd583-592e-4fee-b0fa-cc18e365b9ab-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-pdrc6\" (UID: \"c61fd583-592e-4fee-b0fa-cc18e365b9ab\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6"
Sep 17 17:42:21.587279 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.587245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c61fd583-592e-4fee-b0fa-cc18e365b9ab-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-pdrc6\" (UID: \"c61fd583-592e-4fee-b0fa-cc18e365b9ab\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6"
Sep 17 17:42:21.587399 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.587322    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjksf\" (UniqueName: \"kubernetes.io/projected/c61fd583-592e-4fee-b0fa-cc18e365b9ab-kube-api-access-kjksf\") pod \"openshift-state-metrics-86c5bc79d6-pdrc6\" (UID: \"c61fd583-592e-4fee-b0fa-cc18e365b9ab\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6"
Sep 17 17:42:21.604847 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.604824    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qfb65"]
Sep 17 17:42:21.607736 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.607711    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.610465 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.610443    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:42:21.610963 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.610940    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:42:21.611069 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.610949    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ql85w\""
Sep 17 17:42:21.614268 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.614246    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:42:21.687605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.687555    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61fde122-4e11-434e-ba57-0cd18b35d076-node-exporter-textfile\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.687605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.687608    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61fde122-4e11-434e-ba57-0cd18b35d076-sys\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.687814 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.687631    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61fde122-4e11-434e-ba57-0cd18b35d076-node-exporter-accelerators-collector-config\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.687814 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.687662    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c61fd583-592e-4fee-b0fa-cc18e365b9ab-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-pdrc6\" (UID: \"c61fd583-592e-4fee-b0fa-cc18e365b9ab\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6"
Sep 17 17:42:21.687814 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.687684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61fde122-4e11-434e-ba57-0cd18b35d076-node-exporter-tls\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.687814 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.687702    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61fde122-4e11-434e-ba57-0cd18b35d076-metrics-client-ca\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.687814 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.687716    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61fde122-4e11-434e-ba57-0cd18b35d076-root\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.687814 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.687737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61fde122-4e11-434e-ba57-0cd18b35d076-node-exporter-wtmp\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.687814 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.687756    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61fde122-4e11-434e-ba57-0cd18b35d076-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.688115 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.687828    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjksf\" (UniqueName: \"kubernetes.io/projected/c61fd583-592e-4fee-b0fa-cc18e365b9ab-kube-api-access-kjksf\") pod \"openshift-state-metrics-86c5bc79d6-pdrc6\" (UID: \"c61fd583-592e-4fee-b0fa-cc18e365b9ab\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6"
Sep 17 17:42:21.688115 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.687874    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c61fd583-592e-4fee-b0fa-cc18e365b9ab-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-pdrc6\" (UID: \"c61fd583-592e-4fee-b0fa-cc18e365b9ab\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6"
Sep 17 17:42:21.688115 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.687902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c61fd583-592e-4fee-b0fa-cc18e365b9ab-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-pdrc6\" (UID: \"c61fd583-592e-4fee-b0fa-cc18e365b9ab\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6"
Sep 17 17:42:21.688115 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.687929    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrp45\" (UniqueName: \"kubernetes.io/projected/61fde122-4e11-434e-ba57-0cd18b35d076-kube-api-access-xrp45\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.688541 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.688520    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c61fd583-592e-4fee-b0fa-cc18e365b9ab-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-pdrc6\" (UID: \"c61fd583-592e-4fee-b0fa-cc18e365b9ab\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6"
Sep 17 17:42:21.690108 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.690088    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c61fd583-592e-4fee-b0fa-cc18e365b9ab-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-pdrc6\" (UID: \"c61fd583-592e-4fee-b0fa-cc18e365b9ab\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6"
Sep 17 17:42:21.690244 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.690225    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c61fd583-592e-4fee-b0fa-cc18e365b9ab-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-pdrc6\" (UID: \"c61fd583-592e-4fee-b0fa-cc18e365b9ab\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6"
Sep 17 17:42:21.710026 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.710004    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjksf\" (UniqueName: \"kubernetes.io/projected/c61fd583-592e-4fee-b0fa-cc18e365b9ab-kube-api-access-kjksf\") pod \"openshift-state-metrics-86c5bc79d6-pdrc6\" (UID: \"c61fd583-592e-4fee-b0fa-cc18e365b9ab\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6"
Sep 17 17:42:21.789111 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.789080    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61fde122-4e11-434e-ba57-0cd18b35d076-metrics-client-ca\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.789214 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.789116    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61fde122-4e11-434e-ba57-0cd18b35d076-root\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.789214 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.789142    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61fde122-4e11-434e-ba57-0cd18b35d076-node-exporter-wtmp\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.789214 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.789166    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61fde122-4e11-434e-ba57-0cd18b35d076-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.789376 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.789205    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61fde122-4e11-434e-ba57-0cd18b35d076-root\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.789376 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.789210    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrp45\" (UniqueName: \"kubernetes.io/projected/61fde122-4e11-434e-ba57-0cd18b35d076-kube-api-access-xrp45\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.789376 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.789267    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61fde122-4e11-434e-ba57-0cd18b35d076-node-exporter-textfile\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.789376 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.789300    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61fde122-4e11-434e-ba57-0cd18b35d076-node-exporter-wtmp\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.789582 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.789402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61fde122-4e11-434e-ba57-0cd18b35d076-sys\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.789582 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.789445    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61fde122-4e11-434e-ba57-0cd18b35d076-node-exporter-accelerators-collector-config\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.789582 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.789494    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61fde122-4e11-434e-ba57-0cd18b35d076-node-exporter-tls\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.789582 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.789503    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61fde122-4e11-434e-ba57-0cd18b35d076-sys\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.789582 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.789534    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61fde122-4e11-434e-ba57-0cd18b35d076-node-exporter-textfile\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.789845 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.789750    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61fde122-4e11-434e-ba57-0cd18b35d076-metrics-client-ca\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.790066 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.790045    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61fde122-4e11-434e-ba57-0cd18b35d076-node-exporter-accelerators-collector-config\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.791623 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.791600    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61fde122-4e11-434e-ba57-0cd18b35d076-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.791712 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.791698    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61fde122-4e11-434e-ba57-0cd18b35d076-node-exporter-tls\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.801871 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.801846    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrp45\" (UniqueName: \"kubernetes.io/projected/61fde122-4e11-434e-ba57-0cd18b35d076-kube-api-access-xrp45\") pod \"node-exporter-qfb65\" (UID: \"61fde122-4e11-434e-ba57-0cd18b35d076\") " pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:21.869675 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.869651    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6"
Sep 17 17:42:21.920421 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:21.920389    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qfb65"
Sep 17 17:42:22.153032 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:22.152949    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6"]
Sep 17 17:42:22.156851 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:42:22.156815    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61fd583_592e_4fee_b0fa_cc18e365b9ab.slice/crio-0f8dc5ef0934656c804c992bb674e4b56bad938c1d4f2069f555225c6cf80c16 WatchSource:0}: Error finding container 0f8dc5ef0934656c804c992bb674e4b56bad938c1d4f2069f555225c6cf80c16: Status 404 returned error can't find the container with id 0f8dc5ef0934656c804c992bb674e4b56bad938c1d4f2069f555225c6cf80c16
Sep 17 17:42:22.876487 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:22.876451    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qfb65" event={"ID":"61fde122-4e11-434e-ba57-0cd18b35d076","Type":"ContainerStarted","Data":"9ac9a62a1116712cffb1a1dff328942eeefa6b840ebc9674a13efe5c985d5b97"}
Sep 17 17:42:22.876933 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:22.876521    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qfb65" event={"ID":"61fde122-4e11-434e-ba57-0cd18b35d076","Type":"ContainerStarted","Data":"27e57af08c0b5345a88a01edcfda335ee5dbdf9ba7ea95c885e5b16e70f7f8c1"}
Sep 17 17:42:22.878946 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:22.878916    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6" event={"ID":"c61fd583-592e-4fee-b0fa-cc18e365b9ab","Type":"ContainerStarted","Data":"e1c3ba09aec4532e38e1e01955888f8d141b842d95167e7660aa84b9f74b08ce"}
Sep 17 17:42:22.879061 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:22.878975    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6" event={"ID":"c61fd583-592e-4fee-b0fa-cc18e365b9ab","Type":"ContainerStarted","Data":"6e428ade75393234dfa8acca514da325ae75829baa9b296467e506c21363b8ba"}
Sep 17 17:42:22.879061 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:22.878996    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6" event={"ID":"c61fd583-592e-4fee-b0fa-cc18e365b9ab","Type":"ContainerStarted","Data":"0f8dc5ef0934656c804c992bb674e4b56bad938c1d4f2069f555225c6cf80c16"}
Sep 17 17:42:23.445829 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.445798    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7f8965fb5-79sgc"]
Sep 17 17:42:23.449069 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.449050    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.452284 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.452262    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 17:42:23.452284 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.452279    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vtzgb\""
Sep 17 17:42:23.452462 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.452361    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 17:42:23.452462 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.452398    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 17:42:23.452462 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.452407    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 17:42:23.452462 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.452405    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 17:42:23.452828 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.452815    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-cmabdgs267gan\""
Sep 17 17:42:23.463755 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.463736    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f8965fb5-79sgc"]
Sep 17 17:42:23.501116 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.501094    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd399548-098b-4070-9532-d8730d35d4d3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.501217 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.501122    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cd399548-098b-4070-9532-d8730d35d4d3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.501217 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.501146    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cd399548-098b-4070-9532-d8730d35d4d3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.501217 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.501193    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cd399548-098b-4070-9532-d8730d35d4d3-secret-thanos-querier-tls\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.501328 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.501236    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd399548-098b-4070-9532-d8730d35d4d3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.501328 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.501254    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r92b\" (UniqueName: \"kubernetes.io/projected/cd399548-098b-4070-9532-d8730d35d4d3-kube-api-access-5r92b\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.501328 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.501276    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd399548-098b-4070-9532-d8730d35d4d3-secret-grpc-tls\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.501328 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.501297    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd399548-098b-4070-9532-d8730d35d4d3-metrics-client-ca\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.601785 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.601760    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cd399548-098b-4070-9532-d8730d35d4d3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.601885 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.601795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cd399548-098b-4070-9532-d8730d35d4d3-secret-thanos-querier-tls\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.601885 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.601818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd399548-098b-4070-9532-d8730d35d4d3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.601964 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.601946    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r92b\" (UniqueName: \"kubernetes.io/projected/cd399548-098b-4070-9532-d8730d35d4d3-kube-api-access-5r92b\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.602012 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.601980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd399548-098b-4070-9532-d8730d35d4d3-secret-grpc-tls\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.602059 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.602013    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd399548-098b-4070-9532-d8730d35d4d3-metrics-client-ca\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.602225 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.602198    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd399548-098b-4070-9532-d8730d35d4d3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.602292 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.602250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cd399548-098b-4070-9532-d8730d35d4d3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.602872 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.602847    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd399548-098b-4070-9532-d8730d35d4d3-metrics-client-ca\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.604608 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.604515    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd399548-098b-4070-9532-d8730d35d4d3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.604608 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.604547    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cd399548-098b-4070-9532-d8730d35d4d3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.604756 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.604744    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cd399548-098b-4070-9532-d8730d35d4d3-secret-thanos-querier-tls\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.604994 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.604973    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd399548-098b-4070-9532-d8730d35d4d3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.604994 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.604988    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cd399548-098b-4070-9532-d8730d35d4d3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.605091 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.605052    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd399548-098b-4070-9532-d8730d35d4d3-secret-grpc-tls\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.611271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.611249    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r92b\" (UniqueName: \"kubernetes.io/projected/cd399548-098b-4070-9532-d8730d35d4d3-kube-api-access-5r92b\") pod \"thanos-querier-7f8965fb5-79sgc\" (UID: \"cd399548-098b-4070-9532-d8730d35d4d3\") " pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.761366 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.761328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:23.884412 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.884342    2576 generic.go:356] "Generic (PLEG): container finished" podID="61fde122-4e11-434e-ba57-0cd18b35d076" containerID="9ac9a62a1116712cffb1a1dff328942eeefa6b840ebc9674a13efe5c985d5b97" exitCode=0
Sep 17 17:42:23.884851 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.884419    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qfb65" event={"ID":"61fde122-4e11-434e-ba57-0cd18b35d076","Type":"ContainerDied","Data":"9ac9a62a1116712cffb1a1dff328942eeefa6b840ebc9674a13efe5c985d5b97"}
Sep 17 17:42:23.886598 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.886552    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6" event={"ID":"c61fd583-592e-4fee-b0fa-cc18e365b9ab","Type":"ContainerStarted","Data":"0e5f6d0a82755bfc3e36a11f26930a8153561ebe968b4a1b537fa10e7fcf572b"}
Sep 17 17:42:23.941147 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:23.940994    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-pdrc6" podStartSLOduration=1.919903345 podStartE2EDuration="2.940974852s" totalImagesPullingTime="1.021071507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:23.94074787 +0000 UTC m=+83.533584852" watchObservedRunningTime="2026-09-17 17:42:23.940974852 +0000 UTC m=+83.533811835"
Sep 17 17:42:24.040940 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:24.040912    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f8965fb5-79sgc"]
Sep 17 17:42:24.044508 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:42:24.044486    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd399548_098b_4070_9532_d8730d35d4d3.slice/crio-bb2389c7e8f50079584f2c1d18c9f4e3a48f16f9c4fa6bdbaf6738cbec84282a WatchSource:0}: Error finding container bb2389c7e8f50079584f2c1d18c9f4e3a48f16f9c4fa6bdbaf6738cbec84282a: Status 404 returned error can't find the container with id bb2389c7e8f50079584f2c1d18c9f4e3a48f16f9c4fa6bdbaf6738cbec84282a
Sep 17 17:42:24.893501 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:24.893467    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qfb65" event={"ID":"61fde122-4e11-434e-ba57-0cd18b35d076","Type":"ContainerStarted","Data":"aef44ca47c603830585356991239a664e15267ff43abf8f8213a48430cb646cf"}
Sep 17 17:42:24.893997 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:24.893529    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qfb65" event={"ID":"61fde122-4e11-434e-ba57-0cd18b35d076","Type":"ContainerStarted","Data":"8a160f23b2ca86b2145b6f7b0891a559db25b348088b07553d92c61fe640cc97"}
Sep 17 17:42:24.894928 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:24.894893    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc" event={"ID":"cd399548-098b-4070-9532-d8730d35d4d3","Type":"ContainerStarted","Data":"bb2389c7e8f50079584f2c1d18c9f4e3a48f16f9c4fa6bdbaf6738cbec84282a"}
Sep 17 17:42:24.918285 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:24.918233    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qfb65" podStartSLOduration=3.040377858 podStartE2EDuration="3.918212359s" totalImagesPullingTime="812.826094ms" totalInitContainerRuntime="65.008407ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:24.916547751 +0000 UTC m=+84.509384731" watchObservedRunningTime="2026-09-17 17:42:24.918212359 +0000 UTC m=+84.511049343"
Sep 17 17:42:25.900857 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:25.900831    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc" event={"ID":"cd399548-098b-4070-9532-d8730d35d4d3","Type":"ContainerStarted","Data":"e5eaa160115a6f7ac8283961e245950ebf86cdd4099d816eacbedb6db2a33d9f"}
Sep 17 17:42:26.231442 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.231411    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-769484567f-xttpl"]
Sep 17 17:42:26.236208 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.236182    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-vjfbj"]
Sep 17 17:42:26.236322 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.236304    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.241270 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.241244    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 17:42:26.241376 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.241362    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:42:26.241430 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.241390    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6kxlr\""
Sep 17 17:42:26.241924 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.241903    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vjfbj"
Sep 17 17:42:26.243977 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.243958    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 17:42:26.244060 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.244016    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 17:42:26.244060 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.244021    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:42:26.245372 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.245355    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 17:42:26.245437 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.245378    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 17:42:26.250396 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.250377    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8rq9x\""
Sep 17 17:42:26.250555 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.250537    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 17:42:26.251319 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.250889    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 17:42:26.252038 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.252022    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-vjfbj"]
Sep 17 17:42:26.253385 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.253350    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-769484567f-xttpl"]
Sep 17 17:42:26.322346 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.322319    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-trusted-ca-bundle\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.322346 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.322348    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-oauth-serving-cert\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.322532 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.322365    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ed16268e-bcee-4fe4-bf62-3761059fdd9c-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-vjfbj\" (UID: \"ed16268e-bcee-4fe4-bf62-3761059fdd9c\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vjfbj"
Sep 17 17:42:26.322532 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.322387    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/761edccb-6719-4179-b7fe-5654f708e3e4-console-serving-cert\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.322532 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.322463    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/761edccb-6719-4179-b7fe-5654f708e3e4-console-oauth-config\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.322532 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.322512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpjvp\" (UniqueName: \"kubernetes.io/projected/761edccb-6719-4179-b7fe-5654f708e3e4-kube-api-access-mpjvp\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.322674 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.322537    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-service-ca\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.322674 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.322587    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg4th\" (UniqueName: \"kubernetes.io/projected/ed16268e-bcee-4fe4-bf62-3761059fdd9c-kube-api-access-qg4th\") pod \"monitoring-plugin-5984459ff8-vjfbj\" (UID: \"ed16268e-bcee-4fe4-bf62-3761059fdd9c\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vjfbj"
Sep 17 17:42:26.322674 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.322610    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-console-config\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.423585 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.423522    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-trusted-ca-bundle\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.423813 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.423592    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-oauth-serving-cert\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.423813 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.423622    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ed16268e-bcee-4fe4-bf62-3761059fdd9c-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-vjfbj\" (UID: \"ed16268e-bcee-4fe4-bf62-3761059fdd9c\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vjfbj"
Sep 17 17:42:26.423813 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.423660    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/761edccb-6719-4179-b7fe-5654f708e3e4-console-serving-cert\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.423813 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.423693    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/761edccb-6719-4179-b7fe-5654f708e3e4-console-oauth-config\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.423813 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.423735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpjvp\" (UniqueName: \"kubernetes.io/projected/761edccb-6719-4179-b7fe-5654f708e3e4-kube-api-access-mpjvp\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.423813 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.423759    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-service-ca\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.423813 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.423776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg4th\" (UniqueName: \"kubernetes.io/projected/ed16268e-bcee-4fe4-bf62-3761059fdd9c-kube-api-access-qg4th\") pod \"monitoring-plugin-5984459ff8-vjfbj\" (UID: \"ed16268e-bcee-4fe4-bf62-3761059fdd9c\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vjfbj"
Sep 17 17:42:26.423813 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.423801    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-console-config\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.424539 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.424488    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-oauth-serving-cert\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.424673 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.424538    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-trusted-ca-bundle\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.424906 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.424876    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-console-config\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.425424 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.425403    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-service-ca\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.426668 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.426627    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/761edccb-6719-4179-b7fe-5654f708e3e4-console-oauth-config\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.426750 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.426678    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ed16268e-bcee-4fe4-bf62-3761059fdd9c-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-vjfbj\" (UID: \"ed16268e-bcee-4fe4-bf62-3761059fdd9c\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vjfbj"
Sep 17 17:42:26.426792 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.426773    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/761edccb-6719-4179-b7fe-5654f708e3e4-console-serving-cert\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.433990 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.433972    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpjvp\" (UniqueName: \"kubernetes.io/projected/761edccb-6719-4179-b7fe-5654f708e3e4-kube-api-access-mpjvp\") pod \"console-769484567f-xttpl\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") " pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.435694 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.435678    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg4th\" (UniqueName: \"kubernetes.io/projected/ed16268e-bcee-4fe4-bf62-3761059fdd9c-kube-api-access-qg4th\") pod \"monitoring-plugin-5984459ff8-vjfbj\" (UID: \"ed16268e-bcee-4fe4-bf62-3761059fdd9c\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vjfbj"
Sep 17 17:42:26.552843 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.552762    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:26.557403 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.557378    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vjfbj"
Sep 17 17:42:26.843498 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.843470    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-vjfbj"]
Sep 17 17:42:26.846603 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:42:26.846540    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded16268e_bcee_4fe4_bf62_3761059fdd9c.slice/crio-47a019273ded8c72135bf648c59112b5d9e8d553814a90b3b3ffb7dde56dd4a2 WatchSource:0}: Error finding container 47a019273ded8c72135bf648c59112b5d9e8d553814a90b3b3ffb7dde56dd4a2: Status 404 returned error can't find the container with id 47a019273ded8c72135bf648c59112b5d9e8d553814a90b3b3ffb7dde56dd4a2
Sep 17 17:42:26.847169 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.847133    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-769484567f-xttpl"]
Sep 17 17:42:26.849676 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:42:26.849652    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761edccb_6719_4179_b7fe_5654f708e3e4.slice/crio-7a763e3d1d04e56af3f6720edeb1e7b47860d6e632e6fe5d71427697691c7636 WatchSource:0}: Error finding container 7a763e3d1d04e56af3f6720edeb1e7b47860d6e632e6fe5d71427697691c7636: Status 404 returned error can't find the container with id 7a763e3d1d04e56af3f6720edeb1e7b47860d6e632e6fe5d71427697691c7636
Sep 17 17:42:26.905902 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.905868    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-769484567f-xttpl" event={"ID":"761edccb-6719-4179-b7fe-5654f708e3e4","Type":"ContainerStarted","Data":"7a763e3d1d04e56af3f6720edeb1e7b47860d6e632e6fe5d71427697691c7636"}
Sep 17 17:42:26.907035 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.907007    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vjfbj" event={"ID":"ed16268e-bcee-4fe4-bf62-3761059fdd9c","Type":"ContainerStarted","Data":"47a019273ded8c72135bf648c59112b5d9e8d553814a90b3b3ffb7dde56dd4a2"}
Sep 17 17:42:26.909762 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.909742    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc" event={"ID":"cd399548-098b-4070-9532-d8730d35d4d3","Type":"ContainerStarted","Data":"7c55cee3db7fc77129646759e4385a0c177293ad122a00fd4857f74d608c7b65"}
Sep 17 17:42:26.909834 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.909790    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc" event={"ID":"cd399548-098b-4070-9532-d8730d35d4d3","Type":"ContainerStarted","Data":"cde05d0dc3b90233b8ba78f31c9bc35bacafd6b29c56cf489c4c62338ab6dbad"}
Sep 17 17:42:26.909834 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.909817    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc" event={"ID":"cd399548-098b-4070-9532-d8730d35d4d3","Type":"ContainerStarted","Data":"bb5c68af42e8eb6fd52682a37266d67adfd3cb03dd98b6d4a91e51e3aacb945d"}
Sep 17 17:42:26.909928 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.909842    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc" event={"ID":"cd399548-098b-4070-9532-d8730d35d4d3","Type":"ContainerStarted","Data":"0902d098d28454a155bfda49136ff1689ec88eea7782e909e196e45f3789eb0f"}
Sep 17 17:42:26.909928 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.909870    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc" event={"ID":"cd399548-098b-4070-9532-d8730d35d4d3","Type":"ContainerStarted","Data":"5ab1d05a6d4e701c6a92aa2db7dd102c50410577baa518c2ce48bd7b7c8c6bd5"}
Sep 17 17:42:26.909928 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.909900    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:26.944775 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:26.944734    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc" podStartSLOduration=1.534247082 podStartE2EDuration="3.944716513s" totalImagesPullingTime="2.410469431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:26.943626803 +0000 UTC m=+86.536463785" watchObservedRunningTime="2026-09-17 17:42:26.944716513 +0000 UTC m=+86.537553495"
Sep 17 17:42:28.921618 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:28.921576    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vjfbj" event={"ID":"ed16268e-bcee-4fe4-bf62-3761059fdd9c","Type":"ContainerStarted","Data":"9882d8cfef72f1af067c3e723426c2a9dc6499ea09e8f4bbdd67ffedc612c5f0"}
Sep 17 17:42:28.922046 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:28.921796    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vjfbj"
Sep 17 17:42:28.928343 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:28.928297    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vjfbj"
Sep 17 17:42:28.941113 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:28.941064    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vjfbj" podStartSLOduration=1.525501108 podStartE2EDuration="2.941045465s" totalImagesPullingTime="1.415544357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:28.940064817 +0000 UTC m=+88.532901800" watchObservedRunningTime="2026-09-17 17:42:28.941045465 +0000 UTC m=+88.533882447"
Sep 17 17:42:31.934357 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:31.934323    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-769484567f-xttpl" event={"ID":"761edccb-6719-4179-b7fe-5654f708e3e4","Type":"ContainerStarted","Data":"371c5c8f513302d97b97c9ebb97f7a2ad62d9a8a37339d4be9601897eabb2b6a"}
Sep 17 17:42:31.956082 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:31.956037    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-769484567f-xttpl" podStartSLOduration=1.747289445 podStartE2EDuration="5.95602136s" totalImagesPullingTime="4.208731915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:31.955205164 +0000 UTC m=+91.548042148" watchObservedRunningTime="2026-09-17 17:42:31.95602136 +0000 UTC m=+91.548858342"
Sep 17 17:42:32.921554 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:32.921525    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7f8965fb5-79sgc"
Sep 17 17:42:36.552997 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:36.552956    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:36.553354 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:36.553023    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:36.557538 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:36.557517    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:36.956107 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:36.956079    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:42:46.855627 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:42:46.855589    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qrrjr"
Sep 17 17:44:00.351436 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.351402    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-769484567f-xttpl"]
Sep 17 17:44:00.484494 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.484458    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b88696f68-kmhzr"]
Sep 17 17:44:00.487342 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.487321    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.500137 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.500117    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b88696f68-kmhzr"]
Sep 17 17:44:00.618734 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.618639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e63773f-d18d-440e-bd0f-0bd614b3f555-console-serving-cert\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.618734 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.618680    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-oauth-serving-cert\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.618734 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.618703    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-trusted-ca-bundle\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.618950 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.618791    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e63773f-d18d-440e-bd0f-0bd614b3f555-console-oauth-config\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.618950 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.618820    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-console-config\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.618950 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.618886    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnqt4\" (UniqueName: \"kubernetes.io/projected/6e63773f-d18d-440e-bd0f-0bd614b3f555-kube-api-access-dnqt4\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.619037 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.618959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-service-ca\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.719891 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.719851    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnqt4\" (UniqueName: \"kubernetes.io/projected/6e63773f-d18d-440e-bd0f-0bd614b3f555-kube-api-access-dnqt4\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.720069 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.719913    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-service-ca\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.720069 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.719946    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e63773f-d18d-440e-bd0f-0bd614b3f555-console-serving-cert\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.720069 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.719964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-oauth-serving-cert\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.720069 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.719981    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-trusted-ca-bundle\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.720069 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.720008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e63773f-d18d-440e-bd0f-0bd614b3f555-console-oauth-config\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.720069 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.720038    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-console-config\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.720684 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.720661    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-oauth-serving-cert\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.720840 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.720820    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-console-config\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.720894 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.720834    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-service-ca\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.721010 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.720993    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-trusted-ca-bundle\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.722518 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.722495    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e63773f-d18d-440e-bd0f-0bd614b3f555-console-serving-cert\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.722623 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.722528    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e63773f-d18d-440e-bd0f-0bd614b3f555-console-oauth-config\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.728485 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.728466    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnqt4\" (UniqueName: \"kubernetes.io/projected/6e63773f-d18d-440e-bd0f-0bd614b3f555-kube-api-access-dnqt4\") pod \"console-6b88696f68-kmhzr\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") " pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:00.795434 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:00.795397    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:01.064849 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:01.064777    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b88696f68-kmhzr"]
Sep 17 17:44:01.067732 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:44:01.067698    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e63773f_d18d_440e_bd0f_0bd614b3f555.slice/crio-2647ca2cfa9f9413b4f4c912688ca78cbf59aa783b3958f4ff47c5bc2bebc681 WatchSource:0}: Error finding container 2647ca2cfa9f9413b4f4c912688ca78cbf59aa783b3958f4ff47c5bc2bebc681: Status 404 returned error can't find the container with id 2647ca2cfa9f9413b4f4c912688ca78cbf59aa783b3958f4ff47c5bc2bebc681
Sep 17 17:44:01.257677 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:01.257637    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b88696f68-kmhzr" event={"ID":"6e63773f-d18d-440e-bd0f-0bd614b3f555","Type":"ContainerStarted","Data":"92972ef8e2d2ebe12799ca7dae3ade495855b8e30a7ede4777b6fe0be923a852"}
Sep 17 17:44:01.257838 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:01.257697    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b88696f68-kmhzr" event={"ID":"6e63773f-d18d-440e-bd0f-0bd614b3f555","Type":"ContainerStarted","Data":"2647ca2cfa9f9413b4f4c912688ca78cbf59aa783b3958f4ff47c5bc2bebc681"}
Sep 17 17:44:01.277702 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:01.277652    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b88696f68-kmhzr" podStartSLOduration=1.277635739 podStartE2EDuration="1.277635739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:01.276297145 +0000 UTC m=+180.869134126" watchObservedRunningTime="2026-09-17 17:44:01.277635739 +0000 UTC m=+180.870472721"
Sep 17 17:44:10.796278 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:10.796237    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:10.796832 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:10.796301    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:10.801044 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:10.801026    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:11.306829 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:11.306804    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:44:25.370087 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.370012    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-769484567f-xttpl" podUID="761edccb-6719-4179-b7fe-5654f708e3e4" containerName="console" containerID="cri-o://371c5c8f513302d97b97c9ebb97f7a2ad62d9a8a37339d4be9601897eabb2b6a" gracePeriod=15
Sep 17 17:44:25.608795 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.608774    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:44:25.703706 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.703620    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/761edccb-6719-4179-b7fe-5654f708e3e4-console-oauth-config\") pod \"761edccb-6719-4179-b7fe-5654f708e3e4\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") "
Sep 17 17:44:25.703706 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.703668    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/761edccb-6719-4179-b7fe-5654f708e3e4-console-serving-cert\") pod \"761edccb-6719-4179-b7fe-5654f708e3e4\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") "
Sep 17 17:44:25.703706 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.703701    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-console-config\") pod \"761edccb-6719-4179-b7fe-5654f708e3e4\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") "
Sep 17 17:44:25.703979 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.703911    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-oauth-serving-cert\") pod \"761edccb-6719-4179-b7fe-5654f708e3e4\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") "
Sep 17 17:44:25.703979 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.703959    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-trusted-ca-bundle\") pod \"761edccb-6719-4179-b7fe-5654f708e3e4\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") "
Sep 17 17:44:25.704081 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.703993    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpjvp\" (UniqueName: \"kubernetes.io/projected/761edccb-6719-4179-b7fe-5654f708e3e4-kube-api-access-mpjvp\") pod \"761edccb-6719-4179-b7fe-5654f708e3e4\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") "
Sep 17 17:44:25.704081 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.704043    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-service-ca\") pod \"761edccb-6719-4179-b7fe-5654f708e3e4\" (UID: \"761edccb-6719-4179-b7fe-5654f708e3e4\") "
Sep 17 17:44:25.704081 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.704067    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-console-config" (OuterVolumeSpecName: "console-config") pod "761edccb-6719-4179-b7fe-5654f708e3e4" (UID: "761edccb-6719-4179-b7fe-5654f708e3e4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:25.704286 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.704264    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-console-config\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:44:25.704347 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.704275    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "761edccb-6719-4179-b7fe-5654f708e3e4" (UID: "761edccb-6719-4179-b7fe-5654f708e3e4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:25.704401 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.704358    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "761edccb-6719-4179-b7fe-5654f708e3e4" (UID: "761edccb-6719-4179-b7fe-5654f708e3e4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:25.704638 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.704613    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-service-ca" (OuterVolumeSpecName: "service-ca") pod "761edccb-6719-4179-b7fe-5654f708e3e4" (UID: "761edccb-6719-4179-b7fe-5654f708e3e4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:44:25.706027 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.705997    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/761edccb-6719-4179-b7fe-5654f708e3e4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "761edccb-6719-4179-b7fe-5654f708e3e4" (UID: "761edccb-6719-4179-b7fe-5654f708e3e4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:25.706127 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.706033    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/761edccb-6719-4179-b7fe-5654f708e3e4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "761edccb-6719-4179-b7fe-5654f708e3e4" (UID: "761edccb-6719-4179-b7fe-5654f708e3e4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:25.706127 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.706047    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761edccb-6719-4179-b7fe-5654f708e3e4-kube-api-access-mpjvp" (OuterVolumeSpecName: "kube-api-access-mpjvp") pod "761edccb-6719-4179-b7fe-5654f708e3e4" (UID: "761edccb-6719-4179-b7fe-5654f708e3e4"). InnerVolumeSpecName "kube-api-access-mpjvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:44:25.805529 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.805483    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-service-ca\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:44:25.805529 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.805524    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/761edccb-6719-4179-b7fe-5654f708e3e4-console-oauth-config\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:44:25.805529 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.805533    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/761edccb-6719-4179-b7fe-5654f708e3e4-console-serving-cert\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:44:25.805529 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.805542    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-oauth-serving-cert\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:44:25.805803 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.805551    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/761edccb-6719-4179-b7fe-5654f708e3e4-trusted-ca-bundle\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:44:25.805803 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:25.805577    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpjvp\" (UniqueName: \"kubernetes.io/projected/761edccb-6719-4179-b7fe-5654f708e3e4-kube-api-access-mpjvp\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:44:26.357360 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:26.357317    2576 generic.go:356] "Generic (PLEG): container finished" podID="761edccb-6719-4179-b7fe-5654f708e3e4" containerID="371c5c8f513302d97b97c9ebb97f7a2ad62d9a8a37339d4be9601897eabb2b6a" exitCode=0
Sep 17 17:44:26.357360 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:26.357367    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-769484567f-xttpl"
Sep 17 17:44:26.357594 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:26.357401    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-769484567f-xttpl" event={"ID":"761edccb-6719-4179-b7fe-5654f708e3e4","Type":"ContainerDied","Data":"371c5c8f513302d97b97c9ebb97f7a2ad62d9a8a37339d4be9601897eabb2b6a"}
Sep 17 17:44:26.357594 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:26.357454    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-769484567f-xttpl" event={"ID":"761edccb-6719-4179-b7fe-5654f708e3e4","Type":"ContainerDied","Data":"7a763e3d1d04e56af3f6720edeb1e7b47860d6e632e6fe5d71427697691c7636"}
Sep 17 17:44:26.357594 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:26.357485    2576 scope.go:117] "RemoveContainer" containerID="371c5c8f513302d97b97c9ebb97f7a2ad62d9a8a37339d4be9601897eabb2b6a"
Sep 17 17:44:26.366223 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:26.366206    2576 scope.go:117] "RemoveContainer" containerID="371c5c8f513302d97b97c9ebb97f7a2ad62d9a8a37339d4be9601897eabb2b6a"
Sep 17 17:44:26.366484 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:44:26.366464    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"371c5c8f513302d97b97c9ebb97f7a2ad62d9a8a37339d4be9601897eabb2b6a\": container with ID starting with 371c5c8f513302d97b97c9ebb97f7a2ad62d9a8a37339d4be9601897eabb2b6a not found: ID does not exist" containerID="371c5c8f513302d97b97c9ebb97f7a2ad62d9a8a37339d4be9601897eabb2b6a"
Sep 17 17:44:26.366544 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:26.366492    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"371c5c8f513302d97b97c9ebb97f7a2ad62d9a8a37339d4be9601897eabb2b6a"} err="failed to get container status \"371c5c8f513302d97b97c9ebb97f7a2ad62d9a8a37339d4be9601897eabb2b6a\": rpc error: code = NotFound desc = could not find container \"371c5c8f513302d97b97c9ebb97f7a2ad62d9a8a37339d4be9601897eabb2b6a\": container with ID starting with 371c5c8f513302d97b97c9ebb97f7a2ad62d9a8a37339d4be9601897eabb2b6a not found: ID does not exist"
Sep 17 17:44:26.379757 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:26.379731    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-769484567f-xttpl"]
Sep 17 17:44:26.383591 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:26.383571    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-769484567f-xttpl"]
Sep 17 17:44:26.536402 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:44:26.536371    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="761edccb-6719-4179-b7fe-5654f708e3e4" path="/var/lib/kubelet/pods/761edccb-6719-4179-b7fe-5654f708e3e4/volumes"
Sep 17 17:45:38.875702 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:38.875622    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zp2z7"]
Sep 17 17:45:38.876182 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:38.875958    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="761edccb-6719-4179-b7fe-5654f708e3e4" containerName="console"
Sep 17 17:45:38.876182 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:38.875972    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="761edccb-6719-4179-b7fe-5654f708e3e4" containerName="console"
Sep 17 17:45:38.876182 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:38.876019    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="761edccb-6719-4179-b7fe-5654f708e3e4" containerName="console"
Sep 17 17:45:38.878885 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:38.878870    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zp2z7"
Sep 17 17:45:38.881509 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:38.881488    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-b2njv\""
Sep 17 17:45:38.881812 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:38.881794    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 17:45:38.881875 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:38.881864    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 17:45:38.882024 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:38.882007    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 17:45:38.894951 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:38.894926    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zp2z7"]
Sep 17 17:45:38.931158 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:38.931129    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfpvh\" (UniqueName: \"kubernetes.io/projected/c5e971d8-e9b4-429d-a205-72dc652b45c0-kube-api-access-qfpvh\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-zp2z7\" (UID: \"c5e971d8-e9b4-429d-a205-72dc652b45c0\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zp2z7"
Sep 17 17:45:38.931306 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:38.931217    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c5e971d8-e9b4-429d-a205-72dc652b45c0-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-zp2z7\" (UID: \"c5e971d8-e9b4-429d-a205-72dc652b45c0\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zp2z7"
Sep 17 17:45:39.032503 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:39.032468    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfpvh\" (UniqueName: \"kubernetes.io/projected/c5e971d8-e9b4-429d-a205-72dc652b45c0-kube-api-access-qfpvh\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-zp2z7\" (UID: \"c5e971d8-e9b4-429d-a205-72dc652b45c0\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zp2z7"
Sep 17 17:45:39.032659 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:39.032528    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c5e971d8-e9b4-429d-a205-72dc652b45c0-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-zp2z7\" (UID: \"c5e971d8-e9b4-429d-a205-72dc652b45c0\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zp2z7"
Sep 17 17:45:39.034908 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:39.034881    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c5e971d8-e9b4-429d-a205-72dc652b45c0-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-zp2z7\" (UID: \"c5e971d8-e9b4-429d-a205-72dc652b45c0\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zp2z7"
Sep 17 17:45:39.047282 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:39.047254    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfpvh\" (UniqueName: \"kubernetes.io/projected/c5e971d8-e9b4-429d-a205-72dc652b45c0-kube-api-access-qfpvh\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-zp2z7\" (UID: \"c5e971d8-e9b4-429d-a205-72dc652b45c0\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zp2z7"
Sep 17 17:45:39.189008 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:39.188916    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zp2z7"
Sep 17 17:45:39.481446 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:39.481362    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zp2z7"]
Sep 17 17:45:39.485314 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:45:39.485284    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e971d8_e9b4_429d_a205_72dc652b45c0.slice/crio-cf390bd8af07386482ac5dfed1afc9d069a4e3a830e3d4a6b9cf66f30b7ef353 WatchSource:0}: Error finding container cf390bd8af07386482ac5dfed1afc9d069a4e3a830e3d4a6b9cf66f30b7ef353: Status 404 returned error can't find the container with id cf390bd8af07386482ac5dfed1afc9d069a4e3a830e3d4a6b9cf66f30b7ef353
Sep 17 17:45:39.614593 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:39.614545    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zp2z7" event={"ID":"c5e971d8-e9b4-429d-a205-72dc652b45c0","Type":"ContainerStarted","Data":"cf390bd8af07386482ac5dfed1afc9d069a4e3a830e3d4a6b9cf66f30b7ef353"}
Sep 17 17:45:44.635463 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:44.635422    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zp2z7" event={"ID":"c5e971d8-e9b4-429d-a205-72dc652b45c0","Type":"ContainerStarted","Data":"0d98b01fd66e96dfc8f9acaff7751d22e595e89d99ff8c337d04c00925af96c3"}
Sep 17 17:45:44.635954 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:44.635583    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zp2z7"
Sep 17 17:45:44.658898 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:45:44.658844    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zp2z7" podStartSLOduration=2.5509312189999997 podStartE2EDuration="6.658825189s" totalImagesPullingTime="4.10789397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:44.656624126 +0000 UTC m=+284.249461101" watchObservedRunningTime="2026-09-17 17:45:44.658825189 +0000 UTC m=+284.251662172"
Sep 17 17:46:02.421320 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:46:02.421289    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/ovn-acl-logging/0.log"
Sep 17 17:46:02.423301 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:46:02.423280    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/ovn-acl-logging/0.log"
Sep 17 17:46:02.424774 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:46:02.424745    2576 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:46:05.640822 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:46:05.640798    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zp2z7"
Sep 17 17:47:07.227686 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.227653    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-6lxgg"]
Sep 17 17:47:07.230016 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.229996    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-6lxgg"
Sep 17 17:47:07.232351 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.232322    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 17:47:07.232529 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.232507    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:47:07.233616 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.233596    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-qkmkd\""
Sep 17 17:47:07.233616 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.233609    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:47:07.241287 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.241263    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-26vpf"]
Sep 17 17:47:07.243741 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.243726    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-26vpf"
Sep 17 17:47:07.244618 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.244596    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-6lxgg"]
Sep 17 17:47:07.246091 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.246070    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-f99tp\""
Sep 17 17:47:07.246232 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.246212    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 17:47:07.257413 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.257388    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-26vpf"]
Sep 17 17:47:07.350107 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.350083    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvnj8\" (UniqueName: \"kubernetes.io/projected/6259cf39-a160-464a-ac9c-707366bc2818-kube-api-access-tvnj8\") pod \"kserve-controller-manager-5c5cdf4877-6lxgg\" (UID: \"6259cf39-a160-464a-ac9c-707366bc2818\") " pod="kserve/kserve-controller-manager-5c5cdf4877-6lxgg"
Sep 17 17:47:07.350220 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.350110    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5457555d-ad07-4268-b61e-00d85a39d35a-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-26vpf\" (UID: \"5457555d-ad07-4268-b61e-00d85a39d35a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-26vpf"
Sep 17 17:47:07.350220 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.350140    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4wtn\" (UniqueName: \"kubernetes.io/projected/5457555d-ad07-4268-b61e-00d85a39d35a-kube-api-access-w4wtn\") pod \"llmisvc-controller-manager-645f9c8fd6-26vpf\" (UID: \"5457555d-ad07-4268-b61e-00d85a39d35a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-26vpf"
Sep 17 17:47:07.350290 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.350226    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6259cf39-a160-464a-ac9c-707366bc2818-cert\") pod \"kserve-controller-manager-5c5cdf4877-6lxgg\" (UID: \"6259cf39-a160-464a-ac9c-707366bc2818\") " pod="kserve/kserve-controller-manager-5c5cdf4877-6lxgg"
Sep 17 17:47:07.451026 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.451002    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvnj8\" (UniqueName: \"kubernetes.io/projected/6259cf39-a160-464a-ac9c-707366bc2818-kube-api-access-tvnj8\") pod \"kserve-controller-manager-5c5cdf4877-6lxgg\" (UID: \"6259cf39-a160-464a-ac9c-707366bc2818\") " pod="kserve/kserve-controller-manager-5c5cdf4877-6lxgg"
Sep 17 17:47:07.451140 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.451031    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5457555d-ad07-4268-b61e-00d85a39d35a-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-26vpf\" (UID: \"5457555d-ad07-4268-b61e-00d85a39d35a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-26vpf"
Sep 17 17:47:07.451140 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.451061    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4wtn\" (UniqueName: \"kubernetes.io/projected/5457555d-ad07-4268-b61e-00d85a39d35a-kube-api-access-w4wtn\") pod \"llmisvc-controller-manager-645f9c8fd6-26vpf\" (UID: \"5457555d-ad07-4268-b61e-00d85a39d35a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-26vpf"
Sep 17 17:47:07.451140 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.451087    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6259cf39-a160-464a-ac9c-707366bc2818-cert\") pod \"kserve-controller-manager-5c5cdf4877-6lxgg\" (UID: \"6259cf39-a160-464a-ac9c-707366bc2818\") " pod="kserve/kserve-controller-manager-5c5cdf4877-6lxgg"
Sep 17 17:47:07.451292 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:47:07.451174    2576 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 17 17:47:07.451292 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:47:07.451175    2576 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 17 17:47:07.451292 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:47:07.451232    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6259cf39-a160-464a-ac9c-707366bc2818-cert podName:6259cf39-a160-464a-ac9c-707366bc2818 nodeName:}" failed. No retries permitted until 2026-09-17 17:47:07.951213549 +0000 UTC m=+367.544050512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6259cf39-a160-464a-ac9c-707366bc2818-cert") pod "kserve-controller-manager-5c5cdf4877-6lxgg" (UID: "6259cf39-a160-464a-ac9c-707366bc2818") : secret "kserve-webhook-server-cert" not found
Sep 17 17:47:07.451407 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:47:07.451299    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5457555d-ad07-4268-b61e-00d85a39d35a-cert podName:5457555d-ad07-4268-b61e-00d85a39d35a nodeName:}" failed. No retries permitted until 2026-09-17 17:47:07.951280563 +0000 UTC m=+367.544117528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5457555d-ad07-4268-b61e-00d85a39d35a-cert") pod "llmisvc-controller-manager-645f9c8fd6-26vpf" (UID: "5457555d-ad07-4268-b61e-00d85a39d35a") : secret "llmisvc-webhook-server-cert" not found
Sep 17 17:47:07.460606 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.460583    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4wtn\" (UniqueName: \"kubernetes.io/projected/5457555d-ad07-4268-b61e-00d85a39d35a-kube-api-access-w4wtn\") pod \"llmisvc-controller-manager-645f9c8fd6-26vpf\" (UID: \"5457555d-ad07-4268-b61e-00d85a39d35a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-26vpf"
Sep 17 17:47:07.460694 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.460678    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvnj8\" (UniqueName: \"kubernetes.io/projected/6259cf39-a160-464a-ac9c-707366bc2818-kube-api-access-tvnj8\") pod \"kserve-controller-manager-5c5cdf4877-6lxgg\" (UID: \"6259cf39-a160-464a-ac9c-707366bc2818\") " pod="kserve/kserve-controller-manager-5c5cdf4877-6lxgg"
Sep 17 17:47:07.953855 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.953826    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5457555d-ad07-4268-b61e-00d85a39d35a-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-26vpf\" (UID: \"5457555d-ad07-4268-b61e-00d85a39d35a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-26vpf"
Sep 17 17:47:07.954015 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.953880    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6259cf39-a160-464a-ac9c-707366bc2818-cert\") pod \"kserve-controller-manager-5c5cdf4877-6lxgg\" (UID: \"6259cf39-a160-464a-ac9c-707366bc2818\") " pod="kserve/kserve-controller-manager-5c5cdf4877-6lxgg"
Sep 17 17:47:07.956223 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.956199    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5457555d-ad07-4268-b61e-00d85a39d35a-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-26vpf\" (UID: \"5457555d-ad07-4268-b61e-00d85a39d35a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-26vpf"
Sep 17 17:47:07.956223 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:07.956217    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6259cf39-a160-464a-ac9c-707366bc2818-cert\") pod \"kserve-controller-manager-5c5cdf4877-6lxgg\" (UID: \"6259cf39-a160-464a-ac9c-707366bc2818\") " pod="kserve/kserve-controller-manager-5c5cdf4877-6lxgg"
Sep 17 17:47:08.143155 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:08.143126    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-6lxgg"
Sep 17 17:47:08.154024 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:08.154004    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-26vpf"
Sep 17 17:47:08.418144 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:08.418028    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-6lxgg"]
Sep 17 17:47:08.420494 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:47:08.420468    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6259cf39_a160_464a_ac9c_707366bc2818.slice/crio-53d55a0efa6606eb0c03a2104f4f5493912b53b9f89f7780622e7121466d7e21 WatchSource:0}: Error finding container 53d55a0efa6606eb0c03a2104f4f5493912b53b9f89f7780622e7121466d7e21: Status 404 returned error can't find the container with id 53d55a0efa6606eb0c03a2104f4f5493912b53b9f89f7780622e7121466d7e21
Sep 17 17:47:08.421841 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:08.421824    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:47:08.431750 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:08.431726    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-26vpf"]
Sep 17 17:47:08.433985 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:47:08.433959    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5457555d_ad07_4268_b61e_00d85a39d35a.slice/crio-4d3e388ef3cdefc91e5ed451a9a0ffd0bfc209f76b9d1ab0195809860dc6735a WatchSource:0}: Error finding container 4d3e388ef3cdefc91e5ed451a9a0ffd0bfc209f76b9d1ab0195809860dc6735a: Status 404 returned error can't find the container with id 4d3e388ef3cdefc91e5ed451a9a0ffd0bfc209f76b9d1ab0195809860dc6735a
Sep 17 17:47:08.943267 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:08.943231    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-26vpf" event={"ID":"5457555d-ad07-4268-b61e-00d85a39d35a","Type":"ContainerStarted","Data":"4d3e388ef3cdefc91e5ed451a9a0ffd0bfc209f76b9d1ab0195809860dc6735a"}
Sep 17 17:47:08.946685 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:08.946659    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-6lxgg" event={"ID":"6259cf39-a160-464a-ac9c-707366bc2818","Type":"ContainerStarted","Data":"53d55a0efa6606eb0c03a2104f4f5493912b53b9f89f7780622e7121466d7e21"}
Sep 17 17:47:11.963036 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:11.962928    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-26vpf" event={"ID":"5457555d-ad07-4268-b61e-00d85a39d35a","Type":"ContainerStarted","Data":"ea5d64aea0012c2c7b2be4c5193a8ec12d36890b54bf083751b7931f61ea118f"}
Sep 17 17:47:11.963036 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:11.963006    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-26vpf"
Sep 17 17:47:11.964680 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:11.964645    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-6lxgg" event={"ID":"6259cf39-a160-464a-ac9c-707366bc2818","Type":"ContainerStarted","Data":"10fbb2fc5da20fd73f6aa6850fab83e794f4d74702ac4c8778ff7a78e9ff88d3"}
Sep 17 17:47:11.964825 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:11.964702    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5c5cdf4877-6lxgg"
Sep 17 17:47:11.980058 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:11.980007    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-26vpf" podStartSLOduration=1.8230836639999999 podStartE2EDuration="4.979990991s" totalImagesPullingTime="3.156907327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:11.979763599 +0000 UTC m=+371.572600581" watchObservedRunningTime="2026-09-17 17:47:11.979990991 +0000 UTC m=+371.572827974"
Sep 17 17:47:11.999230 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:11.999189    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5c5cdf4877-6lxgg" podStartSLOduration=1.83168422 podStartE2EDuration="4.999175687s" totalImagesPullingTime="3.167491467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:11.998808895 +0000 UTC m=+371.591645880" watchObservedRunningTime="2026-09-17 17:47:11.999175687 +0000 UTC m=+371.592012670"
Sep 17 17:47:22.170749 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:22.170714    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b88696f68-kmhzr"]
Sep 17 17:47:42.971210 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:42.971180    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-26vpf"
Sep 17 17:47:42.974057 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:42.974037    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5c5cdf4877-6lxgg"
Sep 17 17:47:47.188840 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.188801    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b88696f68-kmhzr" podUID="6e63773f-d18d-440e-bd0f-0bd614b3f555" containerName="console" containerID="cri-o://92972ef8e2d2ebe12799ca7dae3ade495855b8e30a7ede4777b6fe0be923a852" gracePeriod=15
Sep 17 17:47:47.429162 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.429139    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:47:47.550034 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.549972    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-service-ca\") pod \"6e63773f-d18d-440e-bd0f-0bd614b3f555\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") "
Sep 17 17:47:47.550034 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.550006    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-oauth-serving-cert\") pod \"6e63773f-d18d-440e-bd0f-0bd614b3f555\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") "
Sep 17 17:47:47.550177 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.550051    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-console-config\") pod \"6e63773f-d18d-440e-bd0f-0bd614b3f555\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") "
Sep 17 17:47:47.550177 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.550165    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-trusted-ca-bundle\") pod \"6e63773f-d18d-440e-bd0f-0bd614b3f555\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") "
Sep 17 17:47:47.550255 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.550202    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e63773f-d18d-440e-bd0f-0bd614b3f555-console-oauth-config\") pod \"6e63773f-d18d-440e-bd0f-0bd614b3f555\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") "
Sep 17 17:47:47.550255 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.550232    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnqt4\" (UniqueName: \"kubernetes.io/projected/6e63773f-d18d-440e-bd0f-0bd614b3f555-kube-api-access-dnqt4\") pod \"6e63773f-d18d-440e-bd0f-0bd614b3f555\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") "
Sep 17 17:47:47.550345 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.550258    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e63773f-d18d-440e-bd0f-0bd614b3f555-console-serving-cert\") pod \"6e63773f-d18d-440e-bd0f-0bd614b3f555\" (UID: \"6e63773f-d18d-440e-bd0f-0bd614b3f555\") "
Sep 17 17:47:47.550470 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.550440    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6e63773f-d18d-440e-bd0f-0bd614b3f555" (UID: "6e63773f-d18d-440e-bd0f-0bd614b3f555"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:47:47.550470 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.550457    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-service-ca" (OuterVolumeSpecName: "service-ca") pod "6e63773f-d18d-440e-bd0f-0bd614b3f555" (UID: "6e63773f-d18d-440e-bd0f-0bd614b3f555"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:47:47.550470 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.550464    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-console-config" (OuterVolumeSpecName: "console-config") pod "6e63773f-d18d-440e-bd0f-0bd614b3f555" (UID: "6e63773f-d18d-440e-bd0f-0bd614b3f555"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:47:47.550687 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.550650    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6e63773f-d18d-440e-bd0f-0bd614b3f555" (UID: "6e63773f-d18d-440e-bd0f-0bd614b3f555"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:47:47.552302 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.552273    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e63773f-d18d-440e-bd0f-0bd614b3f555-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6e63773f-d18d-440e-bd0f-0bd614b3f555" (UID: "6e63773f-d18d-440e-bd0f-0bd614b3f555"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:47:47.552302 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.552294    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e63773f-d18d-440e-bd0f-0bd614b3f555-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6e63773f-d18d-440e-bd0f-0bd614b3f555" (UID: "6e63773f-d18d-440e-bd0f-0bd614b3f555"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:47:47.552427 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.552326    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e63773f-d18d-440e-bd0f-0bd614b3f555-kube-api-access-dnqt4" (OuterVolumeSpecName: "kube-api-access-dnqt4") pod "6e63773f-d18d-440e-bd0f-0bd614b3f555" (UID: "6e63773f-d18d-440e-bd0f-0bd614b3f555"). InnerVolumeSpecName "kube-api-access-dnqt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:47:47.651282 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.651257    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-service-ca\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:47:47.651282 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.651277    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-oauth-serving-cert\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:47:47.651414 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.651287    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-console-config\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:47:47.651414 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.651295    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e63773f-d18d-440e-bd0f-0bd614b3f555-trusted-ca-bundle\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:47:47.651414 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.651303    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e63773f-d18d-440e-bd0f-0bd614b3f555-console-oauth-config\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:47:47.651414 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.651311    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnqt4\" (UniqueName: \"kubernetes.io/projected/6e63773f-d18d-440e-bd0f-0bd614b3f555-kube-api-access-dnqt4\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:47:47.651414 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:47.651321    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e63773f-d18d-440e-bd0f-0bd614b3f555-console-serving-cert\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:47:48.101743 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:48.101707    2576 generic.go:356] "Generic (PLEG): container finished" podID="6e63773f-d18d-440e-bd0f-0bd614b3f555" containerID="92972ef8e2d2ebe12799ca7dae3ade495855b8e30a7ede4777b6fe0be923a852" exitCode=0
Sep 17 17:47:48.101923 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:48.101768    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b88696f68-kmhzr"
Sep 17 17:47:48.101923 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:48.101794    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b88696f68-kmhzr" event={"ID":"6e63773f-d18d-440e-bd0f-0bd614b3f555","Type":"ContainerDied","Data":"92972ef8e2d2ebe12799ca7dae3ade495855b8e30a7ede4777b6fe0be923a852"}
Sep 17 17:47:48.101923 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:48.101866    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b88696f68-kmhzr" event={"ID":"6e63773f-d18d-440e-bd0f-0bd614b3f555","Type":"ContainerDied","Data":"2647ca2cfa9f9413b4f4c912688ca78cbf59aa783b3958f4ff47c5bc2bebc681"}
Sep 17 17:47:48.101923 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:48.101887    2576 scope.go:117] "RemoveContainer" containerID="92972ef8e2d2ebe12799ca7dae3ade495855b8e30a7ede4777b6fe0be923a852"
Sep 17 17:47:48.110320 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:48.110305    2576 scope.go:117] "RemoveContainer" containerID="92972ef8e2d2ebe12799ca7dae3ade495855b8e30a7ede4777b6fe0be923a852"
Sep 17 17:47:48.110607 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:47:48.110584    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92972ef8e2d2ebe12799ca7dae3ade495855b8e30a7ede4777b6fe0be923a852\": container with ID starting with 92972ef8e2d2ebe12799ca7dae3ade495855b8e30a7ede4777b6fe0be923a852 not found: ID does not exist" containerID="92972ef8e2d2ebe12799ca7dae3ade495855b8e30a7ede4777b6fe0be923a852"
Sep 17 17:47:48.110740 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:48.110613    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92972ef8e2d2ebe12799ca7dae3ade495855b8e30a7ede4777b6fe0be923a852"} err="failed to get container status \"92972ef8e2d2ebe12799ca7dae3ade495855b8e30a7ede4777b6fe0be923a852\": rpc error: code = NotFound desc = could not find container \"92972ef8e2d2ebe12799ca7dae3ade495855b8e30a7ede4777b6fe0be923a852\": container with ID starting with 92972ef8e2d2ebe12799ca7dae3ade495855b8e30a7ede4777b6fe0be923a852 not found: ID does not exist"
Sep 17 17:47:48.122277 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:48.122257    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b88696f68-kmhzr"]
Sep 17 17:47:48.126754 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:48.126733    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b88696f68-kmhzr"]
Sep 17 17:47:48.537816 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:48.537737    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e63773f-d18d-440e-bd0f-0bd614b3f555" path="/var/lib/kubelet/pods/6e63773f-d18d-440e-bd0f-0bd614b3f555/volumes"
Sep 17 17:47:51.724490 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:51.724456    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-6lxgg"]
Sep 17 17:47:51.724969 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:51.724654    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-5c5cdf4877-6lxgg" podUID="6259cf39-a160-464a-ac9c-707366bc2818" containerName="manager" containerID="cri-o://10fbb2fc5da20fd73f6aa6850fab83e794f4d74702ac4c8778ff7a78e9ff88d3" gracePeriod=10
Sep 17 17:47:51.749748 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:51.749723    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-hh4ws"]
Sep 17 17:47:51.750006 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:51.749994    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e63773f-d18d-440e-bd0f-0bd614b3f555" containerName="console"
Sep 17 17:47:51.750052 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:51.750007    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e63773f-d18d-440e-bd0f-0bd614b3f555" containerName="console"
Sep 17 17:47:51.750085 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:51.750061    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e63773f-d18d-440e-bd0f-0bd614b3f555" containerName="console"
Sep 17 17:47:51.751745 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:51.751728    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-hh4ws"
Sep 17 17:47:51.763579 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:51.763542    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-hh4ws"]
Sep 17 17:47:51.882429 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:51.882386    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsgts\" (UniqueName: \"kubernetes.io/projected/8226ce9a-a40d-4986-baf4-bd198131b36e-kube-api-access-qsgts\") pod \"kserve-controller-manager-5c5cdf4877-hh4ws\" (UID: \"8226ce9a-a40d-4986-baf4-bd198131b36e\") " pod="kserve/kserve-controller-manager-5c5cdf4877-hh4ws"
Sep 17 17:47:51.882429 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:51.882432    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8226ce9a-a40d-4986-baf4-bd198131b36e-cert\") pod \"kserve-controller-manager-5c5cdf4877-hh4ws\" (UID: \"8226ce9a-a40d-4986-baf4-bd198131b36e\") " pod="kserve/kserve-controller-manager-5c5cdf4877-hh4ws"
Sep 17 17:47:51.956303 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:51.956281    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-6lxgg"
Sep 17 17:47:51.983306 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:51.983245    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsgts\" (UniqueName: \"kubernetes.io/projected/8226ce9a-a40d-4986-baf4-bd198131b36e-kube-api-access-qsgts\") pod \"kserve-controller-manager-5c5cdf4877-hh4ws\" (UID: \"8226ce9a-a40d-4986-baf4-bd198131b36e\") " pod="kserve/kserve-controller-manager-5c5cdf4877-hh4ws"
Sep 17 17:47:51.983434 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:51.983301    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8226ce9a-a40d-4986-baf4-bd198131b36e-cert\") pod \"kserve-controller-manager-5c5cdf4877-hh4ws\" (UID: \"8226ce9a-a40d-4986-baf4-bd198131b36e\") " pod="kserve/kserve-controller-manager-5c5cdf4877-hh4ws"
Sep 17 17:47:51.985552 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:51.985527    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8226ce9a-a40d-4986-baf4-bd198131b36e-cert\") pod \"kserve-controller-manager-5c5cdf4877-hh4ws\" (UID: \"8226ce9a-a40d-4986-baf4-bd198131b36e\") " pod="kserve/kserve-controller-manager-5c5cdf4877-hh4ws"
Sep 17 17:47:51.992256 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:51.992237    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsgts\" (UniqueName: \"kubernetes.io/projected/8226ce9a-a40d-4986-baf4-bd198131b36e-kube-api-access-qsgts\") pod \"kserve-controller-manager-5c5cdf4877-hh4ws\" (UID: \"8226ce9a-a40d-4986-baf4-bd198131b36e\") " pod="kserve/kserve-controller-manager-5c5cdf4877-hh4ws"
Sep 17 17:47:52.084042 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:52.084013    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvnj8\" (UniqueName: \"kubernetes.io/projected/6259cf39-a160-464a-ac9c-707366bc2818-kube-api-access-tvnj8\") pod \"6259cf39-a160-464a-ac9c-707366bc2818\" (UID: \"6259cf39-a160-464a-ac9c-707366bc2818\") "
Sep 17 17:47:52.084166 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:52.084074    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6259cf39-a160-464a-ac9c-707366bc2818-cert\") pod \"6259cf39-a160-464a-ac9c-707366bc2818\" (UID: \"6259cf39-a160-464a-ac9c-707366bc2818\") "
Sep 17 17:47:52.085973 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:52.085944    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6259cf39-a160-464a-ac9c-707366bc2818-kube-api-access-tvnj8" (OuterVolumeSpecName: "kube-api-access-tvnj8") pod "6259cf39-a160-464a-ac9c-707366bc2818" (UID: "6259cf39-a160-464a-ac9c-707366bc2818"). InnerVolumeSpecName "kube-api-access-tvnj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:47:52.085973 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:52.085954    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6259cf39-a160-464a-ac9c-707366bc2818-cert" (OuterVolumeSpecName: "cert") pod "6259cf39-a160-464a-ac9c-707366bc2818" (UID: "6259cf39-a160-464a-ac9c-707366bc2818"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:47:52.101133 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:52.101116    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-hh4ws"
Sep 17 17:47:52.119518 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:52.119491    2576 generic.go:356] "Generic (PLEG): container finished" podID="6259cf39-a160-464a-ac9c-707366bc2818" containerID="10fbb2fc5da20fd73f6aa6850fab83e794f4d74702ac4c8778ff7a78e9ff88d3" exitCode=0
Sep 17 17:47:52.119634 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:52.119533    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-6lxgg" event={"ID":"6259cf39-a160-464a-ac9c-707366bc2818","Type":"ContainerDied","Data":"10fbb2fc5da20fd73f6aa6850fab83e794f4d74702ac4c8778ff7a78e9ff88d3"}
Sep 17 17:47:52.119634 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:52.119540    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-6lxgg"
Sep 17 17:47:52.119634 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:52.119601    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-6lxgg" event={"ID":"6259cf39-a160-464a-ac9c-707366bc2818","Type":"ContainerDied","Data":"53d55a0efa6606eb0c03a2104f4f5493912b53b9f89f7780622e7121466d7e21"}
Sep 17 17:47:52.119634 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:52.119632    2576 scope.go:117] "RemoveContainer" containerID="10fbb2fc5da20fd73f6aa6850fab83e794f4d74702ac4c8778ff7a78e9ff88d3"
Sep 17 17:47:52.128391 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:52.128377    2576 scope.go:117] "RemoveContainer" containerID="10fbb2fc5da20fd73f6aa6850fab83e794f4d74702ac4c8778ff7a78e9ff88d3"
Sep 17 17:47:52.128712 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:47:52.128689    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10fbb2fc5da20fd73f6aa6850fab83e794f4d74702ac4c8778ff7a78e9ff88d3\": container with ID starting with 10fbb2fc5da20fd73f6aa6850fab83e794f4d74702ac4c8778ff7a78e9ff88d3 not found: ID does not exist" containerID="10fbb2fc5da20fd73f6aa6850fab83e794f4d74702ac4c8778ff7a78e9ff88d3"
Sep 17 17:47:52.128793 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:52.128717    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10fbb2fc5da20fd73f6aa6850fab83e794f4d74702ac4c8778ff7a78e9ff88d3"} err="failed to get container status \"10fbb2fc5da20fd73f6aa6850fab83e794f4d74702ac4c8778ff7a78e9ff88d3\": rpc error: code = NotFound desc = could not find container \"10fbb2fc5da20fd73f6aa6850fab83e794f4d74702ac4c8778ff7a78e9ff88d3\": container with ID starting with 10fbb2fc5da20fd73f6aa6850fab83e794f4d74702ac4c8778ff7a78e9ff88d3 not found: ID does not exist"
Sep 17 17:47:52.141662 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:52.141639    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-6lxgg"]
Sep 17 17:47:52.147086 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:52.147057    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-6lxgg"]
Sep 17 17:47:52.185014 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:52.184992    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvnj8\" (UniqueName: \"kubernetes.io/projected/6259cf39-a160-464a-ac9c-707366bc2818-kube-api-access-tvnj8\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:47:52.185014 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:52.185014    2576 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6259cf39-a160-464a-ac9c-707366bc2818-cert\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:47:52.373543 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:52.373516    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-hh4ws"]
Sep 17 17:47:52.376310 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:47:52.376286    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8226ce9a_a40d_4986_baf4_bd198131b36e.slice/crio-5cc8ba37bb639871039e099c62f1babcf4b39843f1c72c0cd0eb6720a4fb5805 WatchSource:0}: Error finding container 5cc8ba37bb639871039e099c62f1babcf4b39843f1c72c0cd0eb6720a4fb5805: Status 404 returned error can't find the container with id 5cc8ba37bb639871039e099c62f1babcf4b39843f1c72c0cd0eb6720a4fb5805
Sep 17 17:47:52.541904 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:52.541878    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6259cf39-a160-464a-ac9c-707366bc2818" path="/var/lib/kubelet/pods/6259cf39-a160-464a-ac9c-707366bc2818/volumes"
Sep 17 17:47:53.126000 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:53.125969    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-hh4ws" event={"ID":"8226ce9a-a40d-4986-baf4-bd198131b36e","Type":"ContainerStarted","Data":"ba68672487c399a322603ad078bfc366e8929d06f15a594b0367b5afe87d734a"}
Sep 17 17:47:53.126543 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:53.126013    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-hh4ws" event={"ID":"8226ce9a-a40d-4986-baf4-bd198131b36e","Type":"ContainerStarted","Data":"5cc8ba37bb639871039e099c62f1babcf4b39843f1c72c0cd0eb6720a4fb5805"}
Sep 17 17:47:53.126543 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:53.126203    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5c5cdf4877-hh4ws"
Sep 17 17:47:53.143935 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:47:53.143889    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5c5cdf4877-hh4ws" podStartSLOduration=1.741330198 podStartE2EDuration="2.143874872s" totalImagesPullingTime="402.544674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:53.142711804 +0000 UTC m=+412.735548785" watchObservedRunningTime="2026-09-17 17:47:53.143874872 +0000 UTC m=+412.736711854"
Sep 17 17:48:24.134424 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:24.134349    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5c5cdf4877-hh4ws"
Sep 17 17:48:25.892655 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:25.892620    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-9lh52"]
Sep 17 17:48:25.893024 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:25.893013    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6259cf39-a160-464a-ac9c-707366bc2818" containerName="manager"
Sep 17 17:48:25.893069 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:25.893029    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6259cf39-a160-464a-ac9c-707366bc2818" containerName="manager"
Sep 17 17:48:25.893119 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:25.893106    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6259cf39-a160-464a-ac9c-707366bc2818" containerName="manager"
Sep 17 17:48:25.896463 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:25.896441    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-qqbzm"]
Sep 17 17:48:25.896590 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:25.896557    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-9lh52"
Sep 17 17:48:25.899300 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:25.899282    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 17:48:25.899544 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:25.899528    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-rx2cx\""
Sep 17 17:48:25.899922 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:25.899905    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-qqbzm"
Sep 17 17:48:25.902211 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:25.902192    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 17:48:25.902329 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:25.902200    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-58js9\""
Sep 17 17:48:25.902395 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:25.902326    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 17:48:25.907306 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:25.907275    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-9lh52"]
Sep 17 17:48:25.916478 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:25.916388    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-qqbzm"]
Sep 17 17:48:25.928396 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:25.928350    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37c415ed-9fcf-4895-b22f-d87508b4f691-cert\") pod \"odh-model-controller-7b966d7984-qqbzm\" (UID: \"37c415ed-9fcf-4895-b22f-d87508b4f691\") " pod="kserve/odh-model-controller-7b966d7984-qqbzm"
Sep 17 17:48:25.928396 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:25.928390    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqgcj\" (UniqueName: \"kubernetes.io/projected/d6970b2e-d104-4dbb-837e-e1c286d9fd77-kube-api-access-qqgcj\") pod \"model-serving-api-7b954ffc95-9lh52\" (UID: \"d6970b2e-d104-4dbb-837e-e1c286d9fd77\") " pod="kserve/model-serving-api-7b954ffc95-9lh52"
Sep 17 17:48:25.928580 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:25.928421    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6970b2e-d104-4dbb-837e-e1c286d9fd77-tls-certs\") pod \"model-serving-api-7b954ffc95-9lh52\" (UID: \"d6970b2e-d104-4dbb-837e-e1c286d9fd77\") " pod="kserve/model-serving-api-7b954ffc95-9lh52"
Sep 17 17:48:25.928580 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:25.928444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk5w7\" (UniqueName: \"kubernetes.io/projected/37c415ed-9fcf-4895-b22f-d87508b4f691-kube-api-access-dk5w7\") pod \"odh-model-controller-7b966d7984-qqbzm\" (UID: \"37c415ed-9fcf-4895-b22f-d87508b4f691\") " pod="kserve/odh-model-controller-7b966d7984-qqbzm"
Sep 17 17:48:25.928580 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:25.928464    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37c415ed-9fcf-4895-b22f-d87508b4f691-metrics-certs\") pod \"odh-model-controller-7b966d7984-qqbzm\" (UID: \"37c415ed-9fcf-4895-b22f-d87508b4f691\") " pod="kserve/odh-model-controller-7b966d7984-qqbzm"
Sep 17 17:48:26.029066 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:26.029040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqgcj\" (UniqueName: \"kubernetes.io/projected/d6970b2e-d104-4dbb-837e-e1c286d9fd77-kube-api-access-qqgcj\") pod \"model-serving-api-7b954ffc95-9lh52\" (UID: \"d6970b2e-d104-4dbb-837e-e1c286d9fd77\") " pod="kserve/model-serving-api-7b954ffc95-9lh52"
Sep 17 17:48:26.029066 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:26.029071    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6970b2e-d104-4dbb-837e-e1c286d9fd77-tls-certs\") pod \"model-serving-api-7b954ffc95-9lh52\" (UID: \"d6970b2e-d104-4dbb-837e-e1c286d9fd77\") " pod="kserve/model-serving-api-7b954ffc95-9lh52"
Sep 17 17:48:26.029263 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:26.029090    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk5w7\" (UniqueName: \"kubernetes.io/projected/37c415ed-9fcf-4895-b22f-d87508b4f691-kube-api-access-dk5w7\") pod \"odh-model-controller-7b966d7984-qqbzm\" (UID: \"37c415ed-9fcf-4895-b22f-d87508b4f691\") " pod="kserve/odh-model-controller-7b966d7984-qqbzm"
Sep 17 17:48:26.029263 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:26.029179    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37c415ed-9fcf-4895-b22f-d87508b4f691-metrics-certs\") pod \"odh-model-controller-7b966d7984-qqbzm\" (UID: \"37c415ed-9fcf-4895-b22f-d87508b4f691\") " pod="kserve/odh-model-controller-7b966d7984-qqbzm"
Sep 17 17:48:26.029263 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:26.029239    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37c415ed-9fcf-4895-b22f-d87508b4f691-cert\") pod \"odh-model-controller-7b966d7984-qqbzm\" (UID: \"37c415ed-9fcf-4895-b22f-d87508b4f691\") " pod="kserve/odh-model-controller-7b966d7984-qqbzm"
Sep 17 17:48:26.029371 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:48:26.029332    2576 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 17:48:26.029410 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:48:26.029369    2576 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 17 17:48:26.029410 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:48:26.029397    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37c415ed-9fcf-4895-b22f-d87508b4f691-cert podName:37c415ed-9fcf-4895-b22f-d87508b4f691 nodeName:}" failed. No retries permitted until 2026-09-17 17:48:26.529379291 +0000 UTC m=+446.122216268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/37c415ed-9fcf-4895-b22f-d87508b4f691-cert") pod "odh-model-controller-7b966d7984-qqbzm" (UID: "37c415ed-9fcf-4895-b22f-d87508b4f691") : secret "odh-model-controller-webhook-cert" not found
Sep 17 17:48:26.029497 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:48:26.029431    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6970b2e-d104-4dbb-837e-e1c286d9fd77-tls-certs podName:d6970b2e-d104-4dbb-837e-e1c286d9fd77 nodeName:}" failed. No retries permitted until 2026-09-17 17:48:26.529415664 +0000 UTC m=+446.122252637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d6970b2e-d104-4dbb-837e-e1c286d9fd77-tls-certs") pod "model-serving-api-7b954ffc95-9lh52" (UID: "d6970b2e-d104-4dbb-837e-e1c286d9fd77") : secret "model-serving-api-tls" not found
Sep 17 17:48:26.031766 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:26.031742    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37c415ed-9fcf-4895-b22f-d87508b4f691-metrics-certs\") pod \"odh-model-controller-7b966d7984-qqbzm\" (UID: \"37c415ed-9fcf-4895-b22f-d87508b4f691\") " pod="kserve/odh-model-controller-7b966d7984-qqbzm"
Sep 17 17:48:26.051749 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:26.051721    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqgcj\" (UniqueName: \"kubernetes.io/projected/d6970b2e-d104-4dbb-837e-e1c286d9fd77-kube-api-access-qqgcj\") pod \"model-serving-api-7b954ffc95-9lh52\" (UID: \"d6970b2e-d104-4dbb-837e-e1c286d9fd77\") " pod="kserve/model-serving-api-7b954ffc95-9lh52"
Sep 17 17:48:26.052005 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:26.051984    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk5w7\" (UniqueName: \"kubernetes.io/projected/37c415ed-9fcf-4895-b22f-d87508b4f691-kube-api-access-dk5w7\") pod \"odh-model-controller-7b966d7984-qqbzm\" (UID: \"37c415ed-9fcf-4895-b22f-d87508b4f691\") " pod="kserve/odh-model-controller-7b966d7984-qqbzm"
Sep 17 17:48:26.533757 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:26.533720    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37c415ed-9fcf-4895-b22f-d87508b4f691-cert\") pod \"odh-model-controller-7b966d7984-qqbzm\" (UID: \"37c415ed-9fcf-4895-b22f-d87508b4f691\") " pod="kserve/odh-model-controller-7b966d7984-qqbzm"
Sep 17 17:48:26.533757 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:26.533759    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6970b2e-d104-4dbb-837e-e1c286d9fd77-tls-certs\") pod \"model-serving-api-7b954ffc95-9lh52\" (UID: \"d6970b2e-d104-4dbb-837e-e1c286d9fd77\") " pod="kserve/model-serving-api-7b954ffc95-9lh52"
Sep 17 17:48:26.536282 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:26.536255    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6970b2e-d104-4dbb-837e-e1c286d9fd77-tls-certs\") pod \"model-serving-api-7b954ffc95-9lh52\" (UID: \"d6970b2e-d104-4dbb-837e-e1c286d9fd77\") " pod="kserve/model-serving-api-7b954ffc95-9lh52"
Sep 17 17:48:26.536282 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:26.536268    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37c415ed-9fcf-4895-b22f-d87508b4f691-cert\") pod \"odh-model-controller-7b966d7984-qqbzm\" (UID: \"37c415ed-9fcf-4895-b22f-d87508b4f691\") " pod="kserve/odh-model-controller-7b966d7984-qqbzm"
Sep 17 17:48:26.811500 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:26.811419    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-9lh52"
Sep 17 17:48:26.819557 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:26.819528    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-qqbzm"
Sep 17 17:48:27.086939 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:27.086876    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-9lh52"]
Sep 17 17:48:27.088924 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:48:27.088898    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6970b2e_d104_4dbb_837e_e1c286d9fd77.slice/crio-041dae2d6191e12d428a775abedf6e89564708ef5b660b52c59e974ae8284166 WatchSource:0}: Error finding container 041dae2d6191e12d428a775abedf6e89564708ef5b660b52c59e974ae8284166: Status 404 returned error can't find the container with id 041dae2d6191e12d428a775abedf6e89564708ef5b660b52c59e974ae8284166
Sep 17 17:48:27.103291 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:27.103267    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-qqbzm"]
Sep 17 17:48:27.106334 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:48:27.106309    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c415ed_9fcf_4895_b22f_d87508b4f691.slice/crio-782e04378905bb4d95dd3529d2a286affbf7e2e23e96acb76746f8b9182ddb95 WatchSource:0}: Error finding container 782e04378905bb4d95dd3529d2a286affbf7e2e23e96acb76746f8b9182ddb95: Status 404 returned error can't find the container with id 782e04378905bb4d95dd3529d2a286affbf7e2e23e96acb76746f8b9182ddb95
Sep 17 17:48:27.255013 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:27.254967    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-9lh52" event={"ID":"d6970b2e-d104-4dbb-837e-e1c286d9fd77","Type":"ContainerStarted","Data":"041dae2d6191e12d428a775abedf6e89564708ef5b660b52c59e974ae8284166"}
Sep 17 17:48:27.256136 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:27.256112    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-qqbzm" event={"ID":"37c415ed-9fcf-4895-b22f-d87508b4f691","Type":"ContainerStarted","Data":"782e04378905bb4d95dd3529d2a286affbf7e2e23e96acb76746f8b9182ddb95"}
Sep 17 17:48:31.275631 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:31.275594    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-qqbzm" event={"ID":"37c415ed-9fcf-4895-b22f-d87508b4f691","Type":"ContainerStarted","Data":"6102fb1220bed82148bab319a34d62cf1ecd271279023d1a201253dbfda64a0b"}
Sep 17 17:48:31.276067 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:31.275801    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-qqbzm"
Sep 17 17:48:31.277097 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:31.277075    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-9lh52" event={"ID":"d6970b2e-d104-4dbb-837e-e1c286d9fd77","Type":"ContainerStarted","Data":"41665074ee497304ba998650dc75af90d281b7b3e04dd0ae36bb81883a1a007a"}
Sep 17 17:48:31.277230 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:31.277213    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-9lh52"
Sep 17 17:48:31.299106 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:31.299068    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-qqbzm" podStartSLOduration=2.861115738 podStartE2EDuration="6.299055863s" totalImagesPullingTime="3.437940125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:31.298221573 +0000 UTC m=+450.891058556" watchObservedRunningTime="2026-09-17 17:48:31.299055863 +0000 UTC m=+450.891892842"
Sep 17 17:48:31.319838 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:31.319794    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-9lh52" podStartSLOduration=2.907903792 podStartE2EDuration="6.319778259s" totalImagesPullingTime="3.411874467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:31.31888338 +0000 UTC m=+450.911720361" watchObservedRunningTime="2026-09-17 17:48:31.319778259 +0000 UTC m=+450.912615242"
Sep 17 17:48:42.283607 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:42.283557    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-qqbzm"
Sep 17 17:48:42.285395 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:48:42.285373    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-9lh52"
Sep 17 17:50:15.508227 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.508197    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"]
Sep 17 17:50:15.514420 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.514391    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:15.517808 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.517787    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gwm88\""
Sep 17 17:50:15.519461 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.519444    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:50:15.519682 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.519668    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 17:50:15.519682 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.519677    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 17:50:15.520810 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.520790    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:50:15.520916 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.520840    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:50:15.524911 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.524894    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:50:15.527766 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.527745    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"]
Sep 17 17:50:15.634249 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.634225    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-xh6xz\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:15.634357 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.634254    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2glt\" (UniqueName: \"kubernetes.io/projected/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-kube-api-access-h2glt\") pod \"isvc-canary-rollback-predictor-76cbc5464-xh6xz\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:15.634357 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.634272    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-xh6xz\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:15.634357 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.634317    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-xh6xz\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:15.634357 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.634346    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-xh6xz\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:15.734723 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.734700    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-xh6xz\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:15.734829 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.734728    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2glt\" (UniqueName: \"kubernetes.io/projected/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-kube-api-access-h2glt\") pod \"isvc-canary-rollback-predictor-76cbc5464-xh6xz\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:15.734829 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.734745    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-xh6xz\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:15.734903 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.734864    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-xh6xz\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:15.734951 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.734933    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-xh6xz\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:15.735275 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.735255    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-76cbc5464-xh6xz\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:15.735542 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.735525    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-76cbc5464-xh6xz\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:15.735645 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.735526    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-76cbc5464-xh6xz\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:15.737131 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.737112    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-proxy-tls\") pod \"isvc-canary-rollback-predictor-76cbc5464-xh6xz\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:15.743948 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.743926    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2glt\" (UniqueName: \"kubernetes.io/projected/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-kube-api-access-h2glt\") pod \"isvc-canary-rollback-predictor-76cbc5464-xh6xz\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:15.825133 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:15.825083    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:16.097139 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:16.097002    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"]
Sep 17 17:50:16.099682 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:50:16.099657    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15635e2_9ee2_4c10_85b2_ef0fd6a8793b.slice/crio-1d95227f85c08cb08f5fadb751b866ce13d76ac2e85552cb8a77e48bba0392b7 WatchSource:0}: Error finding container 1d95227f85c08cb08f5fadb751b866ce13d76ac2e85552cb8a77e48bba0392b7: Status 404 returned error can't find the container with id 1d95227f85c08cb08f5fadb751b866ce13d76ac2e85552cb8a77e48bba0392b7
Sep 17 17:50:16.673724 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:16.673686    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" event={"ID":"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b","Type":"ContainerStarted","Data":"1d95227f85c08cb08f5fadb751b866ce13d76ac2e85552cb8a77e48bba0392b7"}
Sep 17 17:50:19.691446 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:19.691409    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" event={"ID":"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b","Type":"ContainerStarted","Data":"e800ebb3436c7f692460ced4004c0c2722b7533371e70c0aadc73bafc6f14e0a"}
Sep 17 17:50:23.708110 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:23.708077    2576 generic.go:356] "Generic (PLEG): container finished" podID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerID="e800ebb3436c7f692460ced4004c0c2722b7533371e70c0aadc73bafc6f14e0a" exitCode=0
Sep 17 17:50:23.708505 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:23.708123    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" event={"ID":"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b","Type":"ContainerDied","Data":"e800ebb3436c7f692460ced4004c0c2722b7533371e70c0aadc73bafc6f14e0a"}
Sep 17 17:50:37.785710 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:37.785678    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" event={"ID":"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b","Type":"ContainerStarted","Data":"d35caf4727c33a16b210b9fc945dac770fcf9f0eed40f543d9a8c9fd31fac1e1"}
Sep 17 17:50:39.797355 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:39.797321    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" event={"ID":"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b","Type":"ContainerStarted","Data":"2a1e678fd53053ba12b6da98cf43278f7b23d4c17a7d4dc68bbe74b04cf3cd72"}
Sep 17 17:50:39.797785 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:39.797613    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:39.797785 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:39.797673    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:39.798745 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:39.798715    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:50:39.827723 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:39.827682    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" podStartSLOduration=2.454092916 podStartE2EDuration="24.827656263s" totalImagesPullingTime="18.740493967s" totalInitContainerRuntime="3.63306938s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:50:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:39.823631732 +0000 UTC m=+579.416468715" watchObservedRunningTime="2026-09-17 17:50:39.827656263 +0000 UTC m=+579.420493229"
Sep 17 17:50:40.802025 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:40.801984    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:50:45.805819 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:45.805789    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:50:45.806261 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:45.806236    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:50:55.806581 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:50:55.806524    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:51:02.449631 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:51:02.449602    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/ovn-acl-logging/0.log"
Sep 17 17:51:02.450069 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:51:02.449845    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/ovn-acl-logging/0.log"
Sep 17 17:51:05.807090 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:51:05.806982    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:51:15.806257 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:51:15.806169    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:51:25.806733 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:51:25.806691    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:51:35.806250 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:51:35.806210    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:51:45.806285 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:51:45.806248    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:51:55.807148 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:51:55.807118    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:52:00.170702 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.170667    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"]
Sep 17 17:52:00.173079 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.173055    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:00.177747 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.177724    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-bbc14-kube-rbac-proxy-sar-config\""
Sep 17 17:52:00.177858 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.177724    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-bbc14-predictor-serving-cert\""
Sep 17 17:52:00.187053 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.186988    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"]
Sep 17 17:52:00.332244 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.332208    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/394b0a12-a94a-4573-b20b-224e0f9ddeb2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:00.332419 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.332257    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dswk\" (UniqueName: \"kubernetes.io/projected/394b0a12-a94a-4573-b20b-224e0f9ddeb2-kube-api-access-2dswk\") pod \"isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:00.332419 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.332286    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/394b0a12-a94a-4573-b20b-224e0f9ddeb2-proxy-tls\") pod \"isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:00.332419 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.332310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/394b0a12-a94a-4573-b20b-224e0f9ddeb2-cabundle-cert\") pod \"isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:00.332419 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.332388    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/394b0a12-a94a-4573-b20b-224e0f9ddeb2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:00.433430 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.433353    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/394b0a12-a94a-4573-b20b-224e0f9ddeb2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:00.433430 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.433394    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/394b0a12-a94a-4573-b20b-224e0f9ddeb2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:00.433430 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.433423    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dswk\" (UniqueName: \"kubernetes.io/projected/394b0a12-a94a-4573-b20b-224e0f9ddeb2-kube-api-access-2dswk\") pod \"isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:00.433729 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.433442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/394b0a12-a94a-4573-b20b-224e0f9ddeb2-proxy-tls\") pod \"isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:00.433729 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.433555    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/394b0a12-a94a-4573-b20b-224e0f9ddeb2-cabundle-cert\") pod \"isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:00.433834 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.433804    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/394b0a12-a94a-4573-b20b-224e0f9ddeb2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:00.434175 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.434153    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/394b0a12-a94a-4573-b20b-224e0f9ddeb2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:00.434252 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.434179    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/394b0a12-a94a-4573-b20b-224e0f9ddeb2-cabundle-cert\") pod \"isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:00.435795 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.435769    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/394b0a12-a94a-4573-b20b-224e0f9ddeb2-proxy-tls\") pod \"isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:00.442867 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.442846    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dswk\" (UniqueName: \"kubernetes.io/projected/394b0a12-a94a-4573-b20b-224e0f9ddeb2-kube-api-access-2dswk\") pod \"isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:00.486343 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.486320    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:00.758142 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:00.758096    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"]
Sep 17 17:52:00.760493 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:52:00.760469    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394b0a12_a94a_4573_b20b_224e0f9ddeb2.slice/crio-0cb10d19d5b1026f8b3e1684342cfbbf17cc1665b9fdd5d24dbfed44698cf932 WatchSource:0}: Error finding container 0cb10d19d5b1026f8b3e1684342cfbbf17cc1665b9fdd5d24dbfed44698cf932: Status 404 returned error can't find the container with id 0cb10d19d5b1026f8b3e1684342cfbbf17cc1665b9fdd5d24dbfed44698cf932
Sep 17 17:52:01.121669 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:01.121635    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" event={"ID":"394b0a12-a94a-4573-b20b-224e0f9ddeb2","Type":"ContainerStarted","Data":"2a364e1cf0b3b0ea3ce2e58f3b15ddde5ecc98d71b8ee1593e2ff5548eb6eadf"}
Sep 17 17:52:01.121850 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:01.121689    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" event={"ID":"394b0a12-a94a-4573-b20b-224e0f9ddeb2","Type":"ContainerStarted","Data":"0cb10d19d5b1026f8b3e1684342cfbbf17cc1665b9fdd5d24dbfed44698cf932"}
Sep 17 17:52:05.139442 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:05.139408    2576 generic.go:356] "Generic (PLEG): container finished" podID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerID="2a364e1cf0b3b0ea3ce2e58f3b15ddde5ecc98d71b8ee1593e2ff5548eb6eadf" exitCode=0
Sep 17 17:52:05.139854 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:05.139475    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" event={"ID":"394b0a12-a94a-4573-b20b-224e0f9ddeb2","Type":"ContainerDied","Data":"2a364e1cf0b3b0ea3ce2e58f3b15ddde5ecc98d71b8ee1593e2ff5548eb6eadf"}
Sep 17 17:52:06.146657 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:06.146623    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" event={"ID":"394b0a12-a94a-4573-b20b-224e0f9ddeb2","Type":"ContainerStarted","Data":"977b7a339ee21e8ea327cc18122dc72fd9ca193d494ce1f90c5d730ca516575f"}
Sep 17 17:52:06.147024 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:06.146682    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" event={"ID":"394b0a12-a94a-4573-b20b-224e0f9ddeb2","Type":"ContainerStarted","Data":"d58f56b38383ad11dc16de4d32105b968bf6a8372690edd8e999241db12b9def"}
Sep 17 17:52:06.147024 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:06.146957    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:06.147109 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:06.147090    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:06.148539 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:06.148508    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:52:06.175712 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:06.172530    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" podStartSLOduration=2.216374425 podStartE2EDuration="6.172505925s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.9561315s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:06.170474958 +0000 UTC m=+665.763311941" watchObservedRunningTime="2026-09-17 17:52:06.172505925 +0000 UTC m=+665.765342907"
Sep 17 17:52:07.152576 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:07.152530    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:52:12.156821 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:12.156794    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:52:12.157419 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:12.157390    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:52:22.157450 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:22.157413    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:52:32.157558 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:32.157509    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:52:42.157760 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:42.157718    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:52:52.157993 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:52:52.157954    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:53:02.157467 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:02.157432    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:53:12.158135 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:12.158096    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:53:22.157711 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:22.157683    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:53:35.134648 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:35.134614    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"]
Sep 17 17:53:35.135021 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:35.134894    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="kserve-container" containerID="cri-o://d35caf4727c33a16b210b9fc945dac770fcf9f0eed40f543d9a8c9fd31fac1e1" gracePeriod=30
Sep 17 17:53:35.135021 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:35.134901    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="kube-rbac-proxy" containerID="cri-o://2a1e678fd53053ba12b6da98cf43278f7b23d4c17a7d4dc68bbe74b04cf3cd72" gracePeriod=30
Sep 17 17:53:35.516590 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:35.516483    2576 generic.go:356] "Generic (PLEG): container finished" podID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerID="2a1e678fd53053ba12b6da98cf43278f7b23d4c17a7d4dc68bbe74b04cf3cd72" exitCode=2
Sep 17 17:53:35.516737 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:35.516583    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" event={"ID":"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b","Type":"ContainerDied","Data":"2a1e678fd53053ba12b6da98cf43278f7b23d4c17a7d4dc68bbe74b04cf3cd72"}
Sep 17 17:53:35.803243 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:35.803150    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 17 17:53:35.806398 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:35.806375    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 17:53:39.972240 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:39.972211    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:53:40.078280 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.078200    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-cabundle-cert\") pod \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") "
Sep 17 17:53:40.078280 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.078243    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2glt\" (UniqueName: \"kubernetes.io/projected/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-kube-api-access-h2glt\") pod \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") "
Sep 17 17:53:40.078280 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.078272    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-kserve-provision-location\") pod \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") "
Sep 17 17:53:40.078619 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.078337    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-kube-rbac-proxy-sar-config\") pod \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") "
Sep 17 17:53:40.078619 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.078395    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-proxy-tls\") pod \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\" (UID: \"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b\") "
Sep 17 17:53:40.078735 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.078693    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" (UID: "a15635e2-9ee2-4c10-85b2-ef0fd6a8793b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:40.078790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.078733    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" (UID: "a15635e2-9ee2-4c10-85b2-ef0fd6a8793b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:40.078790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.078773    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" (UID: "a15635e2-9ee2-4c10-85b2-ef0fd6a8793b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:40.080520 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.080490    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" (UID: "a15635e2-9ee2-4c10-85b2-ef0fd6a8793b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:53:40.080738 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.080713    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-kube-api-access-h2glt" (OuterVolumeSpecName: "kube-api-access-h2glt") pod "a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" (UID: "a15635e2-9ee2-4c10-85b2-ef0fd6a8793b"). InnerVolumeSpecName "kube-api-access-h2glt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:53:40.152471 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.152435    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"]
Sep 17 17:53:40.152765 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.152751    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="kube-rbac-proxy"
Sep 17 17:53:40.152765 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.152765    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="kube-rbac-proxy"
Sep 17 17:53:40.152861 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.152783    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="kserve-container"
Sep 17 17:53:40.152861 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.152788    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="kserve-container"
Sep 17 17:53:40.152861 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.152798    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="storage-initializer"
Sep 17 17:53:40.152861 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.152805    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="storage-initializer"
Sep 17 17:53:40.152861 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.152856    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="kserve-container"
Sep 17 17:53:40.153032 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.152867    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerName="kube-rbac-proxy"
Sep 17 17:53:40.154678 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.154659    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"
Sep 17 17:53:40.158005 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.157969    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-bbc14-serving-cert\""
Sep 17 17:53:40.158241 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.158219    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-bbc14-kube-rbac-proxy-sar-config\""
Sep 17 17:53:40.167506 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.167477    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"]
Sep 17 17:53:40.179220 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.179185    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11132c15-bb6c-4157-b02a-9d54d724384e-proxy-tls\") pod \"model-chainer-raw-bbc14-985d5bb68-js5x4\" (UID: \"11132c15-bb6c-4157-b02a-9d54d724384e\") " pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"
Sep 17 17:53:40.179220 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.179223    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11132c15-bb6c-4157-b02a-9d54d724384e-openshift-service-ca-bundle\") pod \"model-chainer-raw-bbc14-985d5bb68-js5x4\" (UID: \"11132c15-bb6c-4157-b02a-9d54d724384e\") " pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"
Sep 17 17:53:40.179429 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.179288    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:53:40.179429 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.179300    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-proxy-tls\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:53:40.179429 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.179310    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-cabundle-cert\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:53:40.179429 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.179318    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2glt\" (UniqueName: \"kubernetes.io/projected/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-kube-api-access-h2glt\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:53:40.179429 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.179325    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b-kserve-provision-location\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:53:40.280437 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.280397    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11132c15-bb6c-4157-b02a-9d54d724384e-proxy-tls\") pod \"model-chainer-raw-bbc14-985d5bb68-js5x4\" (UID: \"11132c15-bb6c-4157-b02a-9d54d724384e\") " pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"
Sep 17 17:53:40.280638 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.280451    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11132c15-bb6c-4157-b02a-9d54d724384e-openshift-service-ca-bundle\") pod \"model-chainer-raw-bbc14-985d5bb68-js5x4\" (UID: \"11132c15-bb6c-4157-b02a-9d54d724384e\") " pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"
Sep 17 17:53:40.280638 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:53:40.280546    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-bbc14-serving-cert: secret "model-chainer-raw-bbc14-serving-cert" not found
Sep 17 17:53:40.280638 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:53:40.280627    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11132c15-bb6c-4157-b02a-9d54d724384e-proxy-tls podName:11132c15-bb6c-4157-b02a-9d54d724384e nodeName:}" failed. No retries permitted until 2026-09-17 17:53:40.780611309 +0000 UTC m=+760.373448269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/11132c15-bb6c-4157-b02a-9d54d724384e-proxy-tls") pod "model-chainer-raw-bbc14-985d5bb68-js5x4" (UID: "11132c15-bb6c-4157-b02a-9d54d724384e") : secret "model-chainer-raw-bbc14-serving-cert" not found
Sep 17 17:53:40.281227 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.281207    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11132c15-bb6c-4157-b02a-9d54d724384e-openshift-service-ca-bundle\") pod \"model-chainer-raw-bbc14-985d5bb68-js5x4\" (UID: \"11132c15-bb6c-4157-b02a-9d54d724384e\") " pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"
Sep 17 17:53:40.541362 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.541326    2576 generic.go:356] "Generic (PLEG): container finished" podID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" containerID="d35caf4727c33a16b210b9fc945dac770fcf9f0eed40f543d9a8c9fd31fac1e1" exitCode=0
Sep 17 17:53:40.541594 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.541382    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"
Sep 17 17:53:40.541594 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.541411    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" event={"ID":"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b","Type":"ContainerDied","Data":"d35caf4727c33a16b210b9fc945dac770fcf9f0eed40f543d9a8c9fd31fac1e1"}
Sep 17 17:53:40.541594 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.541488    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz" event={"ID":"a15635e2-9ee2-4c10-85b2-ef0fd6a8793b","Type":"ContainerDied","Data":"1d95227f85c08cb08f5fadb751b866ce13d76ac2e85552cb8a77e48bba0392b7"}
Sep 17 17:53:40.541594 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.541526    2576 scope.go:117] "RemoveContainer" containerID="2a1e678fd53053ba12b6da98cf43278f7b23d4c17a7d4dc68bbe74b04cf3cd72"
Sep 17 17:53:40.551938 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.551917    2576 scope.go:117] "RemoveContainer" containerID="d35caf4727c33a16b210b9fc945dac770fcf9f0eed40f543d9a8c9fd31fac1e1"
Sep 17 17:53:40.559279 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.559259    2576 scope.go:117] "RemoveContainer" containerID="e800ebb3436c7f692460ced4004c0c2722b7533371e70c0aadc73bafc6f14e0a"
Sep 17 17:53:40.566592 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.566549    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"]
Sep 17 17:53:40.567804 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.567787    2576 scope.go:117] "RemoveContainer" containerID="2a1e678fd53053ba12b6da98cf43278f7b23d4c17a7d4dc68bbe74b04cf3cd72"
Sep 17 17:53:40.568121 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:53:40.568104    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a1e678fd53053ba12b6da98cf43278f7b23d4c17a7d4dc68bbe74b04cf3cd72\": container with ID starting with 2a1e678fd53053ba12b6da98cf43278f7b23d4c17a7d4dc68bbe74b04cf3cd72 not found: ID does not exist" containerID="2a1e678fd53053ba12b6da98cf43278f7b23d4c17a7d4dc68bbe74b04cf3cd72"
Sep 17 17:53:40.568195 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.568133    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a1e678fd53053ba12b6da98cf43278f7b23d4c17a7d4dc68bbe74b04cf3cd72"} err="failed to get container status \"2a1e678fd53053ba12b6da98cf43278f7b23d4c17a7d4dc68bbe74b04cf3cd72\": rpc error: code = NotFound desc = could not find container \"2a1e678fd53053ba12b6da98cf43278f7b23d4c17a7d4dc68bbe74b04cf3cd72\": container with ID starting with 2a1e678fd53053ba12b6da98cf43278f7b23d4c17a7d4dc68bbe74b04cf3cd72 not found: ID does not exist"
Sep 17 17:53:40.568195 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.568158    2576 scope.go:117] "RemoveContainer" containerID="d35caf4727c33a16b210b9fc945dac770fcf9f0eed40f543d9a8c9fd31fac1e1"
Sep 17 17:53:40.568474 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:53:40.568457    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35caf4727c33a16b210b9fc945dac770fcf9f0eed40f543d9a8c9fd31fac1e1\": container with ID starting with d35caf4727c33a16b210b9fc945dac770fcf9f0eed40f543d9a8c9fd31fac1e1 not found: ID does not exist" containerID="d35caf4727c33a16b210b9fc945dac770fcf9f0eed40f543d9a8c9fd31fac1e1"
Sep 17 17:53:40.568547 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.568481    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35caf4727c33a16b210b9fc945dac770fcf9f0eed40f543d9a8c9fd31fac1e1"} err="failed to get container status \"d35caf4727c33a16b210b9fc945dac770fcf9f0eed40f543d9a8c9fd31fac1e1\": rpc error: code = NotFound desc = could not find container \"d35caf4727c33a16b210b9fc945dac770fcf9f0eed40f543d9a8c9fd31fac1e1\": container with ID starting with d35caf4727c33a16b210b9fc945dac770fcf9f0eed40f543d9a8c9fd31fac1e1 not found: ID does not exist"
Sep 17 17:53:40.568547 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.568500    2576 scope.go:117] "RemoveContainer" containerID="e800ebb3436c7f692460ced4004c0c2722b7533371e70c0aadc73bafc6f14e0a"
Sep 17 17:53:40.568804 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:53:40.568787    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e800ebb3436c7f692460ced4004c0c2722b7533371e70c0aadc73bafc6f14e0a\": container with ID starting with e800ebb3436c7f692460ced4004c0c2722b7533371e70c0aadc73bafc6f14e0a not found: ID does not exist" containerID="e800ebb3436c7f692460ced4004c0c2722b7533371e70c0aadc73bafc6f14e0a"
Sep 17 17:53:40.568857 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.568809    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e800ebb3436c7f692460ced4004c0c2722b7533371e70c0aadc73bafc6f14e0a"} err="failed to get container status \"e800ebb3436c7f692460ced4004c0c2722b7533371e70c0aadc73bafc6f14e0a\": rpc error: code = NotFound desc = could not find container \"e800ebb3436c7f692460ced4004c0c2722b7533371e70c0aadc73bafc6f14e0a\": container with ID starting with e800ebb3436c7f692460ced4004c0c2722b7533371e70c0aadc73bafc6f14e0a not found: ID does not exist"
Sep 17 17:53:40.571651 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.571631    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-76cbc5464-xh6xz"]
Sep 17 17:53:40.785390 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.785351    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11132c15-bb6c-4157-b02a-9d54d724384e-proxy-tls\") pod \"model-chainer-raw-bbc14-985d5bb68-js5x4\" (UID: \"11132c15-bb6c-4157-b02a-9d54d724384e\") " pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"
Sep 17 17:53:40.787898 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:40.787871    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11132c15-bb6c-4157-b02a-9d54d724384e-proxy-tls\") pod \"model-chainer-raw-bbc14-985d5bb68-js5x4\" (UID: \"11132c15-bb6c-4157-b02a-9d54d724384e\") " pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"
Sep 17 17:53:41.065424 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:41.065374    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"
Sep 17 17:53:41.334873 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:41.334844    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"]
Sep 17 17:53:41.337765 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:53:41.337740    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11132c15_bb6c_4157_b02a_9d54d724384e.slice/crio-edcce03d8b202ac9f5702ffaf25b3e951567979d39d0c0dd6503919d5d08f7c6 WatchSource:0}: Error finding container edcce03d8b202ac9f5702ffaf25b3e951567979d39d0c0dd6503919d5d08f7c6: Status 404 returned error can't find the container with id edcce03d8b202ac9f5702ffaf25b3e951567979d39d0c0dd6503919d5d08f7c6
Sep 17 17:53:41.339777 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:41.339760    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:53:41.548039 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:41.548001    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4" event={"ID":"11132c15-bb6c-4157-b02a-9d54d724384e","Type":"ContainerStarted","Data":"edcce03d8b202ac9f5702ffaf25b3e951567979d39d0c0dd6503919d5d08f7c6"}
Sep 17 17:53:42.539726 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:42.539695    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a15635e2-9ee2-4c10-85b2-ef0fd6a8793b" path="/var/lib/kubelet/pods/a15635e2-9ee2-4c10-85b2-ef0fd6a8793b/volumes"
Sep 17 17:53:43.557935 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:43.557847    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4" event={"ID":"11132c15-bb6c-4157-b02a-9d54d724384e","Type":"ContainerStarted","Data":"089ff53229758746e7ea8d9d31bc51620e13667e84f2557291bc56868e94d133"}
Sep 17 17:53:43.557935 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:43.557934    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"
Sep 17 17:53:43.575135 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:43.575086    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4" podStartSLOduration=1.745415154 podStartE2EDuration="3.575071093s" totalImagesPullingTime="1.829655939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:43.573478386 +0000 UTC m=+763.166315369" watchObservedRunningTime="2026-09-17 17:53:43.575071093 +0000 UTC m=+763.167908075"
Sep 17 17:53:49.566745 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:49.566717    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"
Sep 17 17:53:50.175285 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.175246    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"]
Sep 17 17:53:50.175450 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.175425    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4" podUID="11132c15-bb6c-4157-b02a-9d54d724384e" containerName="model-chainer-raw-bbc14" containerID="cri-o://089ff53229758746e7ea8d9d31bc51620e13667e84f2557291bc56868e94d133" gracePeriod=30
Sep 17 17:53:50.327861 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.327829    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"]
Sep 17 17:53:50.328100 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.328069    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="kserve-container" containerID="cri-o://d58f56b38383ad11dc16de4d32105b968bf6a8372690edd8e999241db12b9def" gracePeriod=30
Sep 17 17:53:50.328248 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.328099    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="kube-rbac-proxy" containerID="cri-o://977b7a339ee21e8ea327cc18122dc72fd9ca193d494ce1f90c5d730ca516575f" gracePeriod=30
Sep 17 17:53:50.569793 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.569758    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"]
Sep 17 17:53:50.573221 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.573203    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:50.575911 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.575893    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-cb1fd-predictor-serving-cert\""
Sep 17 17:53:50.576001 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.575910    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-cb1fd-kube-rbac-proxy-sar-config\""
Sep 17 17:53:50.589790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.589606    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"]
Sep 17 17:53:50.589922 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.589887    2576 generic.go:356] "Generic (PLEG): container finished" podID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerID="977b7a339ee21e8ea327cc18122dc72fd9ca193d494ce1f90c5d730ca516575f" exitCode=2
Sep 17 17:53:50.590051 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.589975    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" event={"ID":"394b0a12-a94a-4573-b20b-224e0f9ddeb2","Type":"ContainerDied","Data":"977b7a339ee21e8ea327cc18122dc72fd9ca193d494ce1f90c5d730ca516575f"}
Sep 17 17:53:50.663496 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.663461    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63146f5e-47b2-4076-8ee7-6f3e34e181c2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:50.663496 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.663503    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63146f5e-47b2-4076-8ee7-6f3e34e181c2-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:50.663718 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.663593    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63146f5e-47b2-4076-8ee7-6f3e34e181c2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:50.663718 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.663621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63146f5e-47b2-4076-8ee7-6f3e34e181c2-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:50.663718 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.663645    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zh4q\" (UniqueName: \"kubernetes.io/projected/63146f5e-47b2-4076-8ee7-6f3e34e181c2-kube-api-access-5zh4q\") pod \"isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:50.667160 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.667132    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"]
Sep 17 17:53:50.670423 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.670402    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:53:50.673236 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.673215    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-cb1fd-predictor-serving-cert\""
Sep 17 17:53:50.673366 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.673348    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-cb1fd-kube-rbac-proxy-sar-config\""
Sep 17 17:53:50.681652 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.681630    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"]
Sep 17 17:53:50.764621 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.764537    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/120ac961-8617-458b-b043-04a937010915-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb\" (UID: \"120ac961-8617-458b-b043-04a937010915\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:53:50.764621 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.764588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63146f5e-47b2-4076-8ee7-6f3e34e181c2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:50.764860 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.764698    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63146f5e-47b2-4076-8ee7-6f3e34e181c2-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:50.764860 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.764778    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/120ac961-8617-458b-b043-04a937010915-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb\" (UID: \"120ac961-8617-458b-b043-04a937010915\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:53:50.764860 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.764813    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63146f5e-47b2-4076-8ee7-6f3e34e181c2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:50.764982 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:53:50.764864    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-cb1fd-predictor-serving-cert" not found
Sep 17 17:53:50.764982 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.764865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/120ac961-8617-458b-b043-04a937010915-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb\" (UID: \"120ac961-8617-458b-b043-04a937010915\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:53:50.764982 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.764912    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63146f5e-47b2-4076-8ee7-6f3e34e181c2-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:50.764982 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:53:50.764928    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63146f5e-47b2-4076-8ee7-6f3e34e181c2-proxy-tls podName:63146f5e-47b2-4076-8ee7-6f3e34e181c2 nodeName:}" failed. No retries permitted until 2026-09-17 17:53:51.264906437 +0000 UTC m=+770.857743397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/63146f5e-47b2-4076-8ee7-6f3e34e181c2-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" (UID: "63146f5e-47b2-4076-8ee7-6f3e34e181c2") : secret "isvc-sklearn-graph-raw-hpa-cb1fd-predictor-serving-cert" not found
Sep 17 17:53:50.764982 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.764973    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hs99\" (UniqueName: \"kubernetes.io/projected/120ac961-8617-458b-b043-04a937010915-kube-api-access-2hs99\") pod \"isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb\" (UID: \"120ac961-8617-458b-b043-04a937010915\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:53:50.765221 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.765003    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zh4q\" (UniqueName: \"kubernetes.io/projected/63146f5e-47b2-4076-8ee7-6f3e34e181c2-kube-api-access-5zh4q\") pod \"isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:50.765221 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.765062    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/120ac961-8617-458b-b043-04a937010915-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb\" (UID: \"120ac961-8617-458b-b043-04a937010915\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:53:50.765221 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.765124    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63146f5e-47b2-4076-8ee7-6f3e34e181c2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:50.765332 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.765314    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63146f5e-47b2-4076-8ee7-6f3e34e181c2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:50.765500 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.765482    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63146f5e-47b2-4076-8ee7-6f3e34e181c2-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:50.773331 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.773305    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zh4q\" (UniqueName: \"kubernetes.io/projected/63146f5e-47b2-4076-8ee7-6f3e34e181c2-kube-api-access-5zh4q\") pod \"isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:50.865977 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.865953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/120ac961-8617-458b-b043-04a937010915-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb\" (UID: \"120ac961-8617-458b-b043-04a937010915\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:53:50.866096 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.866004    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/120ac961-8617-458b-b043-04a937010915-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb\" (UID: \"120ac961-8617-458b-b043-04a937010915\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:53:50.866096 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.866023    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/120ac961-8617-458b-b043-04a937010915-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb\" (UID: \"120ac961-8617-458b-b043-04a937010915\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:53:50.866096 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.866046    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hs99\" (UniqueName: \"kubernetes.io/projected/120ac961-8617-458b-b043-04a937010915-kube-api-access-2hs99\") pod \"isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb\" (UID: \"120ac961-8617-458b-b043-04a937010915\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:53:50.866096 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.866085    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/120ac961-8617-458b-b043-04a937010915-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb\" (UID: \"120ac961-8617-458b-b043-04a937010915\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:53:50.866327 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:53:50.866131    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-cb1fd-predictor-serving-cert" not found
Sep 17 17:53:50.866327 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:53:50.866194    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/120ac961-8617-458b-b043-04a937010915-proxy-tls podName:120ac961-8617-458b-b043-04a937010915 nodeName:}" failed. No retries permitted until 2026-09-17 17:53:51.366176393 +0000 UTC m=+770.959013360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/120ac961-8617-458b-b043-04a937010915-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" (UID: "120ac961-8617-458b-b043-04a937010915") : secret "isvc-xgboost-graph-raw-hpa-cb1fd-predictor-serving-cert" not found
Sep 17 17:53:50.866455 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.866425    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/120ac961-8617-458b-b043-04a937010915-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb\" (UID: \"120ac961-8617-458b-b043-04a937010915\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:53:50.866720 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.866697    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/120ac961-8617-458b-b043-04a937010915-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb\" (UID: \"120ac961-8617-458b-b043-04a937010915\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:53:50.866795 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.866733    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/120ac961-8617-458b-b043-04a937010915-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb\" (UID: \"120ac961-8617-458b-b043-04a937010915\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:53:50.874808 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:50.874788    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hs99\" (UniqueName: \"kubernetes.io/projected/120ac961-8617-458b-b043-04a937010915-kube-api-access-2hs99\") pod \"isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb\" (UID: \"120ac961-8617-458b-b043-04a937010915\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:53:51.270350 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:51.270320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63146f5e-47b2-4076-8ee7-6f3e34e181c2-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:51.272667 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:51.272643    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63146f5e-47b2-4076-8ee7-6f3e34e181c2-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:51.371205 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:51.371179    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/120ac961-8617-458b-b043-04a937010915-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb\" (UID: \"120ac961-8617-458b-b043-04a937010915\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:53:51.373368 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:51.373346    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/120ac961-8617-458b-b043-04a937010915-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb\" (UID: \"120ac961-8617-458b-b043-04a937010915\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:53:51.484345 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:51.484320    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:51.583055 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:51.583026    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:53:51.760095 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:51.760027    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"]
Sep 17 17:53:51.762599 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:53:51.762577    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63146f5e_47b2_4076_8ee7_6f3e34e181c2.slice/crio-c6f730438b61643d06093357e711a057b87ad5d94cebf012033e0086ef05a4e4 WatchSource:0}: Error finding container c6f730438b61643d06093357e711a057b87ad5d94cebf012033e0086ef05a4e4: Status 404 returned error can't find the container with id c6f730438b61643d06093357e711a057b87ad5d94cebf012033e0086ef05a4e4
Sep 17 17:53:51.856332 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:51.856308    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"]
Sep 17 17:53:51.858039 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:53:51.858013    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod120ac961_8617_458b_b043_04a937010915.slice/crio-1793d6d153406c5135aebdbb62e80426244b87313e6aaf143ffdc4f69a4d67cc WatchSource:0}: Error finding container 1793d6d153406c5135aebdbb62e80426244b87313e6aaf143ffdc4f69a4d67cc: Status 404 returned error can't find the container with id 1793d6d153406c5135aebdbb62e80426244b87313e6aaf143ffdc4f69a4d67cc
Sep 17 17:53:52.152955 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:52.152916    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 17:53:52.157364 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:52.157329    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 17:53:52.600161 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:52.600125    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" event={"ID":"120ac961-8617-458b-b043-04a937010915","Type":"ContainerStarted","Data":"4929a47f52ccd1733216c134db29c67886880d11045488f6dc76d356a702152e"}
Sep 17 17:53:52.600605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:52.600192    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" event={"ID":"120ac961-8617-458b-b043-04a937010915","Type":"ContainerStarted","Data":"1793d6d153406c5135aebdbb62e80426244b87313e6aaf143ffdc4f69a4d67cc"}
Sep 17 17:53:52.601738 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:52.601714    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" event={"ID":"63146f5e-47b2-4076-8ee7-6f3e34e181c2","Type":"ContainerStarted","Data":"24e1135835592106e2dbc06a0e5e4617a25ffb5f96f8512a856d69e12f38ef2b"}
Sep 17 17:53:52.601844 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:52.601758    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" event={"ID":"63146f5e-47b2-4076-8ee7-6f3e34e181c2","Type":"ContainerStarted","Data":"c6f730438b61643d06093357e711a057b87ad5d94cebf012033e0086ef05a4e4"}
Sep 17 17:53:54.565932 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:54.565894    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4" podUID="11132c15-bb6c-4157-b02a-9d54d724384e" containerName="model-chainer-raw-bbc14" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:53:54.771807 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:54.771784    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:53:54.895966 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:54.895940    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/394b0a12-a94a-4573-b20b-224e0f9ddeb2-proxy-tls\") pod \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") "
Sep 17 17:53:54.896111 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:54.895988    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/394b0a12-a94a-4573-b20b-224e0f9ddeb2-kube-rbac-proxy-sar-config\") pod \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") "
Sep 17 17:53:54.896111 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:54.896013    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dswk\" (UniqueName: \"kubernetes.io/projected/394b0a12-a94a-4573-b20b-224e0f9ddeb2-kube-api-access-2dswk\") pod \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") "
Sep 17 17:53:54.896111 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:54.896049    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/394b0a12-a94a-4573-b20b-224e0f9ddeb2-cabundle-cert\") pod \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") "
Sep 17 17:53:54.896111 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:54.896071    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/394b0a12-a94a-4573-b20b-224e0f9ddeb2-kserve-provision-location\") pod \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\" (UID: \"394b0a12-a94a-4573-b20b-224e0f9ddeb2\") "
Sep 17 17:53:54.896465 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:54.896422    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394b0a12-a94a-4573-b20b-224e0f9ddeb2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "394b0a12-a94a-4573-b20b-224e0f9ddeb2" (UID: "394b0a12-a94a-4573-b20b-224e0f9ddeb2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:54.896598 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:54.896432    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394b0a12-a94a-4573-b20b-224e0f9ddeb2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "394b0a12-a94a-4573-b20b-224e0f9ddeb2" (UID: "394b0a12-a94a-4573-b20b-224e0f9ddeb2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:54.896598 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:54.896443    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394b0a12-a94a-4573-b20b-224e0f9ddeb2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "394b0a12-a94a-4573-b20b-224e0f9ddeb2" (UID: "394b0a12-a94a-4573-b20b-224e0f9ddeb2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:54.898094 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:54.898064    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394b0a12-a94a-4573-b20b-224e0f9ddeb2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "394b0a12-a94a-4573-b20b-224e0f9ddeb2" (UID: "394b0a12-a94a-4573-b20b-224e0f9ddeb2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:53:54.898180 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:54.898137    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394b0a12-a94a-4573-b20b-224e0f9ddeb2-kube-api-access-2dswk" (OuterVolumeSpecName: "kube-api-access-2dswk") pod "394b0a12-a94a-4573-b20b-224e0f9ddeb2" (UID: "394b0a12-a94a-4573-b20b-224e0f9ddeb2"). InnerVolumeSpecName "kube-api-access-2dswk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:53:54.997056 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:54.997026    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/394b0a12-a94a-4573-b20b-224e0f9ddeb2-kserve-provision-location\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:53:54.997056 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:54.997054    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/394b0a12-a94a-4573-b20b-224e0f9ddeb2-proxy-tls\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:53:54.997056 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:54.997065    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/394b0a12-a94a-4573-b20b-224e0f9ddeb2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:53:54.997214 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:54.997075    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dswk\" (UniqueName: \"kubernetes.io/projected/394b0a12-a94a-4573-b20b-224e0f9ddeb2-kube-api-access-2dswk\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:53:54.997214 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:54.997083    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/394b0a12-a94a-4573-b20b-224e0f9ddeb2-cabundle-cert\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:53:55.616078 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:55.616049    2576 generic.go:356] "Generic (PLEG): container finished" podID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerID="d58f56b38383ad11dc16de4d32105b968bf6a8372690edd8e999241db12b9def" exitCode=0
Sep 17 17:53:55.616446 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:55.616102    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"
Sep 17 17:53:55.616446 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:55.616129    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" event={"ID":"394b0a12-a94a-4573-b20b-224e0f9ddeb2","Type":"ContainerDied","Data":"d58f56b38383ad11dc16de4d32105b968bf6a8372690edd8e999241db12b9def"}
Sep 17 17:53:55.616446 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:55.616203    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z" event={"ID":"394b0a12-a94a-4573-b20b-224e0f9ddeb2","Type":"ContainerDied","Data":"0cb10d19d5b1026f8b3e1684342cfbbf17cc1665b9fdd5d24dbfed44698cf932"}
Sep 17 17:53:55.616446 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:55.616243    2576 scope.go:117] "RemoveContainer" containerID="977b7a339ee21e8ea327cc18122dc72fd9ca193d494ce1f90c5d730ca516575f"
Sep 17 17:53:55.617771 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:55.617730    2576 generic.go:356] "Generic (PLEG): container finished" podID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerID="24e1135835592106e2dbc06a0e5e4617a25ffb5f96f8512a856d69e12f38ef2b" exitCode=0
Sep 17 17:53:55.617890 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:55.617781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" event={"ID":"63146f5e-47b2-4076-8ee7-6f3e34e181c2","Type":"ContainerDied","Data":"24e1135835592106e2dbc06a0e5e4617a25ffb5f96f8512a856d69e12f38ef2b"}
Sep 17 17:53:55.642777 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:55.642762    2576 scope.go:117] "RemoveContainer" containerID="d58f56b38383ad11dc16de4d32105b968bf6a8372690edd8e999241db12b9def"
Sep 17 17:53:55.660209 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:55.660008    2576 scope.go:117] "RemoveContainer" containerID="2a364e1cf0b3b0ea3ce2e58f3b15ddde5ecc98d71b8ee1593e2ff5548eb6eadf"
Sep 17 17:53:55.668201 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:55.668178    2576 scope.go:117] "RemoveContainer" containerID="977b7a339ee21e8ea327cc18122dc72fd9ca193d494ce1f90c5d730ca516575f"
Sep 17 17:53:55.668545 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:53:55.668521    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"977b7a339ee21e8ea327cc18122dc72fd9ca193d494ce1f90c5d730ca516575f\": container with ID starting with 977b7a339ee21e8ea327cc18122dc72fd9ca193d494ce1f90c5d730ca516575f not found: ID does not exist" containerID="977b7a339ee21e8ea327cc18122dc72fd9ca193d494ce1f90c5d730ca516575f"
Sep 17 17:53:55.668653 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:55.668556    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"977b7a339ee21e8ea327cc18122dc72fd9ca193d494ce1f90c5d730ca516575f"} err="failed to get container status \"977b7a339ee21e8ea327cc18122dc72fd9ca193d494ce1f90c5d730ca516575f\": rpc error: code = NotFound desc = could not find container \"977b7a339ee21e8ea327cc18122dc72fd9ca193d494ce1f90c5d730ca516575f\": container with ID starting with 977b7a339ee21e8ea327cc18122dc72fd9ca193d494ce1f90c5d730ca516575f not found: ID does not exist"
Sep 17 17:53:55.668653 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:55.668626    2576 scope.go:117] "RemoveContainer" containerID="d58f56b38383ad11dc16de4d32105b968bf6a8372690edd8e999241db12b9def"
Sep 17 17:53:55.669171 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:53:55.669071    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58f56b38383ad11dc16de4d32105b968bf6a8372690edd8e999241db12b9def\": container with ID starting with d58f56b38383ad11dc16de4d32105b968bf6a8372690edd8e999241db12b9def not found: ID does not exist" containerID="d58f56b38383ad11dc16de4d32105b968bf6a8372690edd8e999241db12b9def"
Sep 17 17:53:55.669283 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:55.669178    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58f56b38383ad11dc16de4d32105b968bf6a8372690edd8e999241db12b9def"} err="failed to get container status \"d58f56b38383ad11dc16de4d32105b968bf6a8372690edd8e999241db12b9def\": rpc error: code = NotFound desc = could not find container \"d58f56b38383ad11dc16de4d32105b968bf6a8372690edd8e999241db12b9def\": container with ID starting with d58f56b38383ad11dc16de4d32105b968bf6a8372690edd8e999241db12b9def not found: ID does not exist"
Sep 17 17:53:55.669283 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:55.669201    2576 scope.go:117] "RemoveContainer" containerID="2a364e1cf0b3b0ea3ce2e58f3b15ddde5ecc98d71b8ee1593e2ff5548eb6eadf"
Sep 17 17:53:55.669530 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:53:55.669509    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a364e1cf0b3b0ea3ce2e58f3b15ddde5ecc98d71b8ee1593e2ff5548eb6eadf\": container with ID starting with 2a364e1cf0b3b0ea3ce2e58f3b15ddde5ecc98d71b8ee1593e2ff5548eb6eadf not found: ID does not exist" containerID="2a364e1cf0b3b0ea3ce2e58f3b15ddde5ecc98d71b8ee1593e2ff5548eb6eadf"
Sep 17 17:53:55.670015 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:55.669984    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a364e1cf0b3b0ea3ce2e58f3b15ddde5ecc98d71b8ee1593e2ff5548eb6eadf"} err="failed to get container status \"2a364e1cf0b3b0ea3ce2e58f3b15ddde5ecc98d71b8ee1593e2ff5548eb6eadf\": rpc error: code = NotFound desc = could not find container \"2a364e1cf0b3b0ea3ce2e58f3b15ddde5ecc98d71b8ee1593e2ff5548eb6eadf\": container with ID starting with 2a364e1cf0b3b0ea3ce2e58f3b15ddde5ecc98d71b8ee1593e2ff5548eb6eadf not found: ID does not exist"
Sep 17 17:53:55.671329 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:55.671287    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"]
Sep 17 17:53:55.673281 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:55.673262    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-bbc14-predictor-75448cb46f-vhx8z"]
Sep 17 17:53:56.537802 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:56.537769    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" path="/var/lib/kubelet/pods/394b0a12-a94a-4573-b20b-224e0f9ddeb2/volumes"
Sep 17 17:53:56.623227 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:56.623195    2576 generic.go:356] "Generic (PLEG): container finished" podID="120ac961-8617-458b-b043-04a937010915" containerID="4929a47f52ccd1733216c134db29c67886880d11045488f6dc76d356a702152e" exitCode=0
Sep 17 17:53:56.623605 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:56.623267    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" event={"ID":"120ac961-8617-458b-b043-04a937010915","Type":"ContainerDied","Data":"4929a47f52ccd1733216c134db29c67886880d11045488f6dc76d356a702152e"}
Sep 17 17:53:56.626516 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:56.626495    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" event={"ID":"63146f5e-47b2-4076-8ee7-6f3e34e181c2","Type":"ContainerStarted","Data":"d4f71f1a397231637793b48a0fd02dee8737b13acd7862ed9d933898ccaa88a7"}
Sep 17 17:53:56.626620 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:56.626540    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" event={"ID":"63146f5e-47b2-4076-8ee7-6f3e34e181c2","Type":"ContainerStarted","Data":"cff9942fb24f6aa64c4adb013db8aa011113325e4c515a28c58a8e8458847034"}
Sep 17 17:53:56.626851 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:56.626834    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:56.626903 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:56.626886    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:53:56.628018 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:56.627997    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 17:53:56.666679 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:56.666638    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" podStartSLOduration=2.90556372 podStartE2EDuration="6.666612823s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.761049103s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:56.665868594 +0000 UTC m=+776.258705580" watchObservedRunningTime="2026-09-17 17:53:56.666612823 +0000 UTC m=+776.259449808"
Sep 17 17:53:57.631575 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:57.631520    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 17:53:59.566858 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:53:59.566819    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4" podUID="11132c15-bb6c-4157-b02a-9d54d724384e" containerName="model-chainer-raw-bbc14" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:54:02.638037 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:02.638004    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:54:02.638669 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:02.638632    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 17:54:04.567174 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:04.567126    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4" podUID="11132c15-bb6c-4157-b02a-9d54d724384e" containerName="model-chainer-raw-bbc14" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:54:04.567652 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:04.567286    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"
Sep 17 17:54:09.566654 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:09.566609    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4" podUID="11132c15-bb6c-4157-b02a-9d54d724384e" containerName="model-chainer-raw-bbc14" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:54:12.638819 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:12.638777    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 17:54:14.566477 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:14.566376    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4" podUID="11132c15-bb6c-4157-b02a-9d54d724384e" containerName="model-chainer-raw-bbc14" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:54:15.719894 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:15.719816    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" event={"ID":"120ac961-8617-458b-b043-04a937010915","Type":"ContainerStarted","Data":"9043cddc2077cd9bce9ecc486d7dd5b894eb8c75453a428a6defe7964dc9ff62"}
Sep 17 17:54:15.719894 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:15.719880    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" event={"ID":"120ac961-8617-458b-b043-04a937010915","Type":"ContainerStarted","Data":"d7c75675e75a10e78bbe94767c9c079b7b9f5971fe784d9ce4c5d07d63a41516"}
Sep 17 17:54:15.720262 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:15.720163    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:54:15.720313 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:15.720295    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:54:15.721414 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:15.721388    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" podUID="120ac961-8617-458b-b043-04a937010915" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 17:54:15.740123 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:15.739957    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" podStartSLOduration=3.170472278 podStartE2EDuration="25.739943208s" totalImagesPullingTime="18.754427832s" totalInitContainerRuntime="3.815043098s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:54:15.739523708 +0000 UTC m=+795.332360692" watchObservedRunningTime="2026-09-17 17:54:15.739943208 +0000 UTC m=+795.332780189"
Sep 17 17:54:16.724734 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:16.724694    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" podUID="120ac961-8617-458b-b043-04a937010915" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 17:54:19.565387 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:19.565352    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4" podUID="11132c15-bb6c-4157-b02a-9d54d724384e" containerName="model-chainer-raw-bbc14" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:54:20.748511 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:20.748473    2576 generic.go:356] "Generic (PLEG): container finished" podID="11132c15-bb6c-4157-b02a-9d54d724384e" containerID="089ff53229758746e7ea8d9d31bc51620e13667e84f2557291bc56868e94d133" exitCode=0
Sep 17 17:54:20.748877 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:20.748533    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4" event={"ID":"11132c15-bb6c-4157-b02a-9d54d724384e","Type":"ContainerDied","Data":"089ff53229758746e7ea8d9d31bc51620e13667e84f2557291bc56868e94d133"}
Sep 17 17:54:20.853202 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:20.853178    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"
Sep 17 17:54:21.018733 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:21.018662    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11132c15-bb6c-4157-b02a-9d54d724384e-proxy-tls\") pod \"11132c15-bb6c-4157-b02a-9d54d724384e\" (UID: \"11132c15-bb6c-4157-b02a-9d54d724384e\") "
Sep 17 17:54:21.018886 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:21.018760    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11132c15-bb6c-4157-b02a-9d54d724384e-openshift-service-ca-bundle\") pod \"11132c15-bb6c-4157-b02a-9d54d724384e\" (UID: \"11132c15-bb6c-4157-b02a-9d54d724384e\") "
Sep 17 17:54:21.019124 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:21.019098    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11132c15-bb6c-4157-b02a-9d54d724384e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "11132c15-bb6c-4157-b02a-9d54d724384e" (UID: "11132c15-bb6c-4157-b02a-9d54d724384e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:54:21.020730 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:21.020707    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11132c15-bb6c-4157-b02a-9d54d724384e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "11132c15-bb6c-4157-b02a-9d54d724384e" (UID: "11132c15-bb6c-4157-b02a-9d54d724384e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:54:21.119883 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:21.119851    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11132c15-bb6c-4157-b02a-9d54d724384e-openshift-service-ca-bundle\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:54:21.119883 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:21.119880    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11132c15-bb6c-4157-b02a-9d54d724384e-proxy-tls\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:54:21.729842 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:21.729815    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:54:21.730319 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:21.730292    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" podUID="120ac961-8617-458b-b043-04a937010915" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 17:54:21.754815 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:21.754791    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"
Sep 17 17:54:21.754815 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:21.754801    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4" event={"ID":"11132c15-bb6c-4157-b02a-9d54d724384e","Type":"ContainerDied","Data":"edcce03d8b202ac9f5702ffaf25b3e951567979d39d0c0dd6503919d5d08f7c6"}
Sep 17 17:54:21.755325 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:21.754862    2576 scope.go:117] "RemoveContainer" containerID="089ff53229758746e7ea8d9d31bc51620e13667e84f2557291bc56868e94d133"
Sep 17 17:54:21.774092 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:21.774066    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"]
Sep 17 17:54:21.777177 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:21.777155    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-bbc14-985d5bb68-js5x4"]
Sep 17 17:54:22.536760 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:22.536731    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11132c15-bb6c-4157-b02a-9d54d724384e" path="/var/lib/kubelet/pods/11132c15-bb6c-4157-b02a-9d54d724384e/volumes"
Sep 17 17:54:22.639285 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:22.639250    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 17:54:31.730518 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:31.730478    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" podUID="120ac961-8617-458b-b043-04a937010915" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 17:54:32.638870 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:32.638828    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 17:54:41.731091 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:41.731054    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" podUID="120ac961-8617-458b-b043-04a937010915" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 17:54:42.639043 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:42.639003    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 17:54:51.730764 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:51.730725    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" podUID="120ac961-8617-458b-b043-04a937010915" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 17:54:52.638766 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:54:52.638725    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 17:55:01.730295 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:01.730259    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" podUID="120ac961-8617-458b-b043-04a937010915" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 17:55:02.638799 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:02.638761    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 17:55:04.533643 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:04.533601    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 17:55:11.730465 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:11.730421    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" podUID="120ac961-8617-458b-b043-04a937010915" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 17:55:14.536962 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:14.536933    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:55:21.730710 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:21.730683    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:55:30.434436 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.434402    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8"]
Sep 17 17:55:30.434895 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.434750    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="kube-rbac-proxy"
Sep 17 17:55:30.434895 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.434761    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="kube-rbac-proxy"
Sep 17 17:55:30.434895 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.434774    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="kserve-container"
Sep 17 17:55:30.434895 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.434779    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="kserve-container"
Sep 17 17:55:30.434895 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.434789    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="storage-initializer"
Sep 17 17:55:30.434895 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.434794    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="storage-initializer"
Sep 17 17:55:30.434895 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.434802    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11132c15-bb6c-4157-b02a-9d54d724384e" containerName="model-chainer-raw-bbc14"
Sep 17 17:55:30.434895 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.434807    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="11132c15-bb6c-4157-b02a-9d54d724384e" containerName="model-chainer-raw-bbc14"
Sep 17 17:55:30.434895 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.434861    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="kserve-container"
Sep 17 17:55:30.434895 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.434870    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="11132c15-bb6c-4157-b02a-9d54d724384e" containerName="model-chainer-raw-bbc14"
Sep 17 17:55:30.434895 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.434878    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="394b0a12-a94a-4573-b20b-224e0f9ddeb2" containerName="kube-rbac-proxy"
Sep 17 17:55:30.437817 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.437797    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8"
Sep 17 17:55:30.441247 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.441223    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-cb1fd-serving-cert\""
Sep 17 17:55:30.441374 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.441258    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-cb1fd-kube-rbac-proxy-sar-config\""
Sep 17 17:55:30.455819 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.455795    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8"]
Sep 17 17:55:30.544618 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.544594    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8831241f-9fea-415f-a060-8953a4cac1cd-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8\" (UID: \"8831241f-9fea-415f-a060-8953a4cac1cd\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8"
Sep 17 17:55:30.544768 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.544633    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8831241f-9fea-415f-a060-8953a4cac1cd-proxy-tls\") pod \"model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8\" (UID: \"8831241f-9fea-415f-a060-8953a4cac1cd\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8"
Sep 17 17:55:30.645429 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.645399    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8831241f-9fea-415f-a060-8953a4cac1cd-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8\" (UID: \"8831241f-9fea-415f-a060-8953a4cac1cd\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8"
Sep 17 17:55:30.645596 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.645441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8831241f-9fea-415f-a060-8953a4cac1cd-proxy-tls\") pod \"model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8\" (UID: \"8831241f-9fea-415f-a060-8953a4cac1cd\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8"
Sep 17 17:55:30.646051 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.646031    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8831241f-9fea-415f-a060-8953a4cac1cd-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8\" (UID: \"8831241f-9fea-415f-a060-8953a4cac1cd\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8"
Sep 17 17:55:30.647815 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.647799    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8831241f-9fea-415f-a060-8953a4cac1cd-proxy-tls\") pod \"model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8\" (UID: \"8831241f-9fea-415f-a060-8953a4cac1cd\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8"
Sep 17 17:55:30.748056 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:30.747976    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8"
Sep 17 17:55:31.019254 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:31.019231    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8"]
Sep 17 17:55:31.021768 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:55:31.021739    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8831241f_9fea_415f_a060_8953a4cac1cd.slice/crio-e712a2e6a38cb48bfc539b9e8785a693d6d77dbf0128c895e38ef1689db875f8 WatchSource:0}: Error finding container e712a2e6a38cb48bfc539b9e8785a693d6d77dbf0128c895e38ef1689db875f8: Status 404 returned error can't find the container with id e712a2e6a38cb48bfc539b9e8785a693d6d77dbf0128c895e38ef1689db875f8
Sep 17 17:55:31.040272 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:31.040247    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8" event={"ID":"8831241f-9fea-415f-a060-8953a4cac1cd","Type":"ContainerStarted","Data":"e712a2e6a38cb48bfc539b9e8785a693d6d77dbf0128c895e38ef1689db875f8"}
Sep 17 17:55:32.048449 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:32.046619    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8" event={"ID":"8831241f-9fea-415f-a060-8953a4cac1cd","Type":"ContainerStarted","Data":"48117f33669576edf78db58e1fbab550091f494acb0a63b1d2326dfcadc4139b"}
Sep 17 17:55:32.048449 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:32.047416    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8"
Sep 17 17:55:32.064729 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:32.064685    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8" podStartSLOduration=2.064670843 podStartE2EDuration="2.064670843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:55:32.063354726 +0000 UTC m=+871.656191708" watchObservedRunningTime="2026-09-17 17:55:32.064670843 +0000 UTC m=+871.657507838"
Sep 17 17:55:39.060432 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:39.060404    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8"
Sep 17 17:55:40.507551 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:40.507523    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8"]
Sep 17 17:55:40.507925 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:40.507702    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8" podUID="8831241f-9fea-415f-a060-8953a4cac1cd" containerName="model-chainer-raw-hpa-cb1fd" containerID="cri-o://48117f33669576edf78db58e1fbab550091f494acb0a63b1d2326dfcadc4139b" gracePeriod=30
Sep 17 17:55:40.624818 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:40.624781    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"]
Sep 17 17:55:40.625081 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:40.625057    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kserve-container" containerID="cri-o://cff9942fb24f6aa64c4adb013db8aa011113325e4c515a28c58a8e8458847034" gracePeriod=30
Sep 17 17:55:40.625192 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:40.625095    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kube-rbac-proxy" containerID="cri-o://d4f71f1a397231637793b48a0fd02dee8737b13acd7862ed9d933898ccaa88a7" gracePeriod=30
Sep 17 17:55:40.719617 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:40.719589    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"]
Sep 17 17:55:40.719840 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:40.719814    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" podUID="120ac961-8617-458b-b043-04a937010915" containerName="kserve-container" containerID="cri-o://d7c75675e75a10e78bbe94767c9c079b7b9f5971fe784d9ce4c5d07d63a41516" gracePeriod=30
Sep 17 17:55:40.719914 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:40.719852    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" podUID="120ac961-8617-458b-b043-04a937010915" containerName="kube-rbac-proxy" containerID="cri-o://9043cddc2077cd9bce9ecc486d7dd5b894eb8c75453a428a6defe7964dc9ff62" gracePeriod=30
Sep 17 17:55:41.087942 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:41.087911    2576 generic.go:356] "Generic (PLEG): container finished" podID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerID="d4f71f1a397231637793b48a0fd02dee8737b13acd7862ed9d933898ccaa88a7" exitCode=2
Sep 17 17:55:41.088125 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:41.087986    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" event={"ID":"63146f5e-47b2-4076-8ee7-6f3e34e181c2","Type":"ContainerDied","Data":"d4f71f1a397231637793b48a0fd02dee8737b13acd7862ed9d933898ccaa88a7"}
Sep 17 17:55:41.090055 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:41.090033    2576 generic.go:356] "Generic (PLEG): container finished" podID="120ac961-8617-458b-b043-04a937010915" containerID="9043cddc2077cd9bce9ecc486d7dd5b894eb8c75453a428a6defe7964dc9ff62" exitCode=2
Sep 17 17:55:41.090167 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:41.090109    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" event={"ID":"120ac961-8617-458b-b043-04a937010915","Type":"ContainerDied","Data":"9043cddc2077cd9bce9ecc486d7dd5b894eb8c75453a428a6defe7964dc9ff62"}
Sep 17 17:55:41.725379 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:41.725339    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" podUID="120ac961-8617-458b-b043-04a937010915" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 17 17:55:41.730859 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:41.730834    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" podUID="120ac961-8617-458b-b043-04a937010915" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 17:55:42.631829 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:42.631786    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 17:55:44.059177 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:44.059132    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8" podUID="8831241f-9fea-415f-a060-8953a4cac1cd" containerName="model-chainer-raw-hpa-cb1fd" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:55:44.533881 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:44.533839    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 17:55:44.972916 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:44.972893    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:55:45.054207 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.054182    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hs99\" (UniqueName: \"kubernetes.io/projected/120ac961-8617-458b-b043-04a937010915-kube-api-access-2hs99\") pod \"120ac961-8617-458b-b043-04a937010915\" (UID: \"120ac961-8617-458b-b043-04a937010915\") "
Sep 17 17:55:45.054386 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.054252    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/120ac961-8617-458b-b043-04a937010915-kube-rbac-proxy-sar-config\") pod \"120ac961-8617-458b-b043-04a937010915\" (UID: \"120ac961-8617-458b-b043-04a937010915\") "
Sep 17 17:55:45.054386 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.054270    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/120ac961-8617-458b-b043-04a937010915-cabundle-cert\") pod \"120ac961-8617-458b-b043-04a937010915\" (UID: \"120ac961-8617-458b-b043-04a937010915\") "
Sep 17 17:55:45.054386 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.054305    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/120ac961-8617-458b-b043-04a937010915-kserve-provision-location\") pod \"120ac961-8617-458b-b043-04a937010915\" (UID: \"120ac961-8617-458b-b043-04a937010915\") "
Sep 17 17:55:45.054386 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.054342    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/120ac961-8617-458b-b043-04a937010915-proxy-tls\") pod \"120ac961-8617-458b-b043-04a937010915\" (UID: \"120ac961-8617-458b-b043-04a937010915\") "
Sep 17 17:55:45.054675 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.054644    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/120ac961-8617-458b-b043-04a937010915-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "120ac961-8617-458b-b043-04a937010915" (UID: "120ac961-8617-458b-b043-04a937010915"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:55:45.054675 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.054656    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/120ac961-8617-458b-b043-04a937010915-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "120ac961-8617-458b-b043-04a937010915" (UID: "120ac961-8617-458b-b043-04a937010915"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:55:45.054675 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.054668    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/120ac961-8617-458b-b043-04a937010915-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "120ac961-8617-458b-b043-04a937010915" (UID: "120ac961-8617-458b-b043-04a937010915"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:55:45.056330 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.056307    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/120ac961-8617-458b-b043-04a937010915-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "120ac961-8617-458b-b043-04a937010915" (UID: "120ac961-8617-458b-b043-04a937010915"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:55:45.056421 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.056399    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/120ac961-8617-458b-b043-04a937010915-kube-api-access-2hs99" (OuterVolumeSpecName: "kube-api-access-2hs99") pod "120ac961-8617-458b-b043-04a937010915" (UID: "120ac961-8617-458b-b043-04a937010915"). InnerVolumeSpecName "kube-api-access-2hs99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:55:45.114789 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.114718    2576 generic.go:356] "Generic (PLEG): container finished" podID="120ac961-8617-458b-b043-04a937010915" containerID="d7c75675e75a10e78bbe94767c9c079b7b9f5971fe784d9ce4c5d07d63a41516" exitCode=0
Sep 17 17:55:45.114789 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.114778    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"
Sep 17 17:55:45.115141 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.114787    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" event={"ID":"120ac961-8617-458b-b043-04a937010915","Type":"ContainerDied","Data":"d7c75675e75a10e78bbe94767c9c079b7b9f5971fe784d9ce4c5d07d63a41516"}
Sep 17 17:55:45.115141 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.114850    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb" event={"ID":"120ac961-8617-458b-b043-04a937010915","Type":"ContainerDied","Data":"1793d6d153406c5135aebdbb62e80426244b87313e6aaf143ffdc4f69a4d67cc"}
Sep 17 17:55:45.115141 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.114876    2576 scope.go:117] "RemoveContainer" containerID="9043cddc2077cd9bce9ecc486d7dd5b894eb8c75453a428a6defe7964dc9ff62"
Sep 17 17:55:45.123223 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.123206    2576 scope.go:117] "RemoveContainer" containerID="d7c75675e75a10e78bbe94767c9c079b7b9f5971fe784d9ce4c5d07d63a41516"
Sep 17 17:55:45.130434 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.130413    2576 scope.go:117] "RemoveContainer" containerID="4929a47f52ccd1733216c134db29c67886880d11045488f6dc76d356a702152e"
Sep 17 17:55:45.137368 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.137348    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"]
Sep 17 17:55:45.138126 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.138109    2576 scope.go:117] "RemoveContainer" containerID="9043cddc2077cd9bce9ecc486d7dd5b894eb8c75453a428a6defe7964dc9ff62"
Sep 17 17:55:45.138380 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:55:45.138362    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9043cddc2077cd9bce9ecc486d7dd5b894eb8c75453a428a6defe7964dc9ff62\": container with ID starting with 9043cddc2077cd9bce9ecc486d7dd5b894eb8c75453a428a6defe7964dc9ff62 not found: ID does not exist" containerID="9043cddc2077cd9bce9ecc486d7dd5b894eb8c75453a428a6defe7964dc9ff62"
Sep 17 17:55:45.138441 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.138392    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9043cddc2077cd9bce9ecc486d7dd5b894eb8c75453a428a6defe7964dc9ff62"} err="failed to get container status \"9043cddc2077cd9bce9ecc486d7dd5b894eb8c75453a428a6defe7964dc9ff62\": rpc error: code = NotFound desc = could not find container \"9043cddc2077cd9bce9ecc486d7dd5b894eb8c75453a428a6defe7964dc9ff62\": container with ID starting with 9043cddc2077cd9bce9ecc486d7dd5b894eb8c75453a428a6defe7964dc9ff62 not found: ID does not exist"
Sep 17 17:55:45.138441 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.138417    2576 scope.go:117] "RemoveContainer" containerID="d7c75675e75a10e78bbe94767c9c079b7b9f5971fe784d9ce4c5d07d63a41516"
Sep 17 17:55:45.138692 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:55:45.138674    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7c75675e75a10e78bbe94767c9c079b7b9f5971fe784d9ce4c5d07d63a41516\": container with ID starting with d7c75675e75a10e78bbe94767c9c079b7b9f5971fe784d9ce4c5d07d63a41516 not found: ID does not exist" containerID="d7c75675e75a10e78bbe94767c9c079b7b9f5971fe784d9ce4c5d07d63a41516"
Sep 17 17:55:45.138731 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.138698    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7c75675e75a10e78bbe94767c9c079b7b9f5971fe784d9ce4c5d07d63a41516"} err="failed to get container status \"d7c75675e75a10e78bbe94767c9c079b7b9f5971fe784d9ce4c5d07d63a41516\": rpc error: code = NotFound desc = could not find container \"d7c75675e75a10e78bbe94767c9c079b7b9f5971fe784d9ce4c5d07d63a41516\": container with ID starting with d7c75675e75a10e78bbe94767c9c079b7b9f5971fe784d9ce4c5d07d63a41516 not found: ID does not exist"
Sep 17 17:55:45.138731 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.138713    2576 scope.go:117] "RemoveContainer" containerID="4929a47f52ccd1733216c134db29c67886880d11045488f6dc76d356a702152e"
Sep 17 17:55:45.138937 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:55:45.138921    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4929a47f52ccd1733216c134db29c67886880d11045488f6dc76d356a702152e\": container with ID starting with 4929a47f52ccd1733216c134db29c67886880d11045488f6dc76d356a702152e not found: ID does not exist" containerID="4929a47f52ccd1733216c134db29c67886880d11045488f6dc76d356a702152e"
Sep 17 17:55:45.138979 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.138941    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4929a47f52ccd1733216c134db29c67886880d11045488f6dc76d356a702152e"} err="failed to get container status \"4929a47f52ccd1733216c134db29c67886880d11045488f6dc76d356a702152e\": rpc error: code = NotFound desc = could not find container \"4929a47f52ccd1733216c134db29c67886880d11045488f6dc76d356a702152e\": container with ID starting with 4929a47f52ccd1733216c134db29c67886880d11045488f6dc76d356a702152e not found: ID does not exist"
Sep 17 17:55:45.141811 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.141787    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-cb1fd-predictor-7b7db6d6f7-5b8nb"]
Sep 17 17:55:45.155619 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.155599    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/120ac961-8617-458b-b043-04a937010915-kserve-provision-location\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:55:45.155703 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.155620    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/120ac961-8617-458b-b043-04a937010915-proxy-tls\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:55:45.155703 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.155636    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hs99\" (UniqueName: \"kubernetes.io/projected/120ac961-8617-458b-b043-04a937010915-kube-api-access-2hs99\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:55:45.155703 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.155644    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/120ac961-8617-458b-b043-04a937010915-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:55:45.155703 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.155652    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/120ac961-8617-458b-b043-04a937010915-cabundle-cert\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:55:45.457917 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.457895    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:55:45.558850 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.558819    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63146f5e-47b2-4076-8ee7-6f3e34e181c2-kserve-provision-location\") pod \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") "
Sep 17 17:55:45.558850 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.558848    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zh4q\" (UniqueName: \"kubernetes.io/projected/63146f5e-47b2-4076-8ee7-6f3e34e181c2-kube-api-access-5zh4q\") pod \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") "
Sep 17 17:55:45.559048 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.558879    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63146f5e-47b2-4076-8ee7-6f3e34e181c2-proxy-tls\") pod \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") "
Sep 17 17:55:45.559048 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.558954    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63146f5e-47b2-4076-8ee7-6f3e34e181c2-cabundle-cert\") pod \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") "
Sep 17 17:55:45.559048 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.558997    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63146f5e-47b2-4076-8ee7-6f3e34e181c2-kube-rbac-proxy-sar-config\") pod \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\" (UID: \"63146f5e-47b2-4076-8ee7-6f3e34e181c2\") "
Sep 17 17:55:45.559237 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.559206    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63146f5e-47b2-4076-8ee7-6f3e34e181c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "63146f5e-47b2-4076-8ee7-6f3e34e181c2" (UID: "63146f5e-47b2-4076-8ee7-6f3e34e181c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:55:45.559370 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.559350    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63146f5e-47b2-4076-8ee7-6f3e34e181c2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "63146f5e-47b2-4076-8ee7-6f3e34e181c2" (UID: "63146f5e-47b2-4076-8ee7-6f3e34e181c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:55:45.559444 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.559419    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63146f5e-47b2-4076-8ee7-6f3e34e181c2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "63146f5e-47b2-4076-8ee7-6f3e34e181c2" (UID: "63146f5e-47b2-4076-8ee7-6f3e34e181c2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:55:45.561016 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.560996    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63146f5e-47b2-4076-8ee7-6f3e34e181c2-kube-api-access-5zh4q" (OuterVolumeSpecName: "kube-api-access-5zh4q") pod "63146f5e-47b2-4076-8ee7-6f3e34e181c2" (UID: "63146f5e-47b2-4076-8ee7-6f3e34e181c2"). InnerVolumeSpecName "kube-api-access-5zh4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:55:45.561087 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.561014    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63146f5e-47b2-4076-8ee7-6f3e34e181c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "63146f5e-47b2-4076-8ee7-6f3e34e181c2" (UID: "63146f5e-47b2-4076-8ee7-6f3e34e181c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:55:45.659710 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.659618    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63146f5e-47b2-4076-8ee7-6f3e34e181c2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:55:45.659710 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.659656    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63146f5e-47b2-4076-8ee7-6f3e34e181c2-kserve-provision-location\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:55:45.659710 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.659666    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zh4q\" (UniqueName: \"kubernetes.io/projected/63146f5e-47b2-4076-8ee7-6f3e34e181c2-kube-api-access-5zh4q\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:55:45.659710 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.659676    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63146f5e-47b2-4076-8ee7-6f3e34e181c2-proxy-tls\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:55:45.659710 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:45.659685    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63146f5e-47b2-4076-8ee7-6f3e34e181c2-cabundle-cert\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:55:46.120666 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:46.120634    2576 generic.go:356] "Generic (PLEG): container finished" podID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerID="cff9942fb24f6aa64c4adb013db8aa011113325e4c515a28c58a8e8458847034" exitCode=0
Sep 17 17:55:46.121107 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:46.120696    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"
Sep 17 17:55:46.121107 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:46.120724    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" event={"ID":"63146f5e-47b2-4076-8ee7-6f3e34e181c2","Type":"ContainerDied","Data":"cff9942fb24f6aa64c4adb013db8aa011113325e4c515a28c58a8e8458847034"}
Sep 17 17:55:46.121107 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:46.120804    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k" event={"ID":"63146f5e-47b2-4076-8ee7-6f3e34e181c2","Type":"ContainerDied","Data":"c6f730438b61643d06093357e711a057b87ad5d94cebf012033e0086ef05a4e4"}
Sep 17 17:55:46.121107 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:46.120843    2576 scope.go:117] "RemoveContainer" containerID="d4f71f1a397231637793b48a0fd02dee8737b13acd7862ed9d933898ccaa88a7"
Sep 17 17:55:46.130135 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:46.130110    2576 scope.go:117] "RemoveContainer" containerID="cff9942fb24f6aa64c4adb013db8aa011113325e4c515a28c58a8e8458847034"
Sep 17 17:55:46.138059 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:46.138042    2576 scope.go:117] "RemoveContainer" containerID="24e1135835592106e2dbc06a0e5e4617a25ffb5f96f8512a856d69e12f38ef2b"
Sep 17 17:55:46.145291 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:46.145271    2576 scope.go:117] "RemoveContainer" containerID="d4f71f1a397231637793b48a0fd02dee8737b13acd7862ed9d933898ccaa88a7"
Sep 17 17:55:46.145595 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:55:46.145555    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4f71f1a397231637793b48a0fd02dee8737b13acd7862ed9d933898ccaa88a7\": container with ID starting with d4f71f1a397231637793b48a0fd02dee8737b13acd7862ed9d933898ccaa88a7 not found: ID does not exist" containerID="d4f71f1a397231637793b48a0fd02dee8737b13acd7862ed9d933898ccaa88a7"
Sep 17 17:55:46.145712 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:46.145600    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f71f1a397231637793b48a0fd02dee8737b13acd7862ed9d933898ccaa88a7"} err="failed to get container status \"d4f71f1a397231637793b48a0fd02dee8737b13acd7862ed9d933898ccaa88a7\": rpc error: code = NotFound desc = could not find container \"d4f71f1a397231637793b48a0fd02dee8737b13acd7862ed9d933898ccaa88a7\": container with ID starting with d4f71f1a397231637793b48a0fd02dee8737b13acd7862ed9d933898ccaa88a7 not found: ID does not exist"
Sep 17 17:55:46.145712 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:46.145618    2576 scope.go:117] "RemoveContainer" containerID="cff9942fb24f6aa64c4adb013db8aa011113325e4c515a28c58a8e8458847034"
Sep 17 17:55:46.145864 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:55:46.145849    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff9942fb24f6aa64c4adb013db8aa011113325e4c515a28c58a8e8458847034\": container with ID starting with cff9942fb24f6aa64c4adb013db8aa011113325e4c515a28c58a8e8458847034 not found: ID does not exist" containerID="cff9942fb24f6aa64c4adb013db8aa011113325e4c515a28c58a8e8458847034"
Sep 17 17:55:46.145908 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:46.145868    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff9942fb24f6aa64c4adb013db8aa011113325e4c515a28c58a8e8458847034"} err="failed to get container status \"cff9942fb24f6aa64c4adb013db8aa011113325e4c515a28c58a8e8458847034\": rpc error: code = NotFound desc = could not find container \"cff9942fb24f6aa64c4adb013db8aa011113325e4c515a28c58a8e8458847034\": container with ID starting with cff9942fb24f6aa64c4adb013db8aa011113325e4c515a28c58a8e8458847034 not found: ID does not exist"
Sep 17 17:55:46.145908 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:46.145881    2576 scope.go:117] "RemoveContainer" containerID="24e1135835592106e2dbc06a0e5e4617a25ffb5f96f8512a856d69e12f38ef2b"
Sep 17 17:55:46.146126 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:55:46.146109    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e1135835592106e2dbc06a0e5e4617a25ffb5f96f8512a856d69e12f38ef2b\": container with ID starting with 24e1135835592106e2dbc06a0e5e4617a25ffb5f96f8512a856d69e12f38ef2b not found: ID does not exist" containerID="24e1135835592106e2dbc06a0e5e4617a25ffb5f96f8512a856d69e12f38ef2b"
Sep 17 17:55:46.146171 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:46.146133    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e1135835592106e2dbc06a0e5e4617a25ffb5f96f8512a856d69e12f38ef2b"} err="failed to get container status \"24e1135835592106e2dbc06a0e5e4617a25ffb5f96f8512a856d69e12f38ef2b\": rpc error: code = NotFound desc = could not find container \"24e1135835592106e2dbc06a0e5e4617a25ffb5f96f8512a856d69e12f38ef2b\": container with ID starting with 24e1135835592106e2dbc06a0e5e4617a25ffb5f96f8512a856d69e12f38ef2b not found: ID does not exist"
Sep 17 17:55:46.150482 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:46.150460    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"]
Sep 17 17:55:46.154544 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:46.154524    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-cb1fd-predictor-b7f956db5-nbz8k"]
Sep 17 17:55:46.537603 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:46.537508    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="120ac961-8617-458b-b043-04a937010915" path="/var/lib/kubelet/pods/120ac961-8617-458b-b043-04a937010915/volumes"
Sep 17 17:55:46.538019 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:46.538006    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" path="/var/lib/kubelet/pods/63146f5e-47b2-4076-8ee7-6f3e34e181c2/volumes"
Sep 17 17:55:49.058719 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:49.058681    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8" podUID="8831241f-9fea-415f-a060-8953a4cac1cd" containerName="model-chainer-raw-hpa-cb1fd" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:55:54.059023 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:54.058979    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8" podUID="8831241f-9fea-415f-a060-8953a4cac1cd" containerName="model-chainer-raw-hpa-cb1fd" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:55:54.059401 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:54.059092    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8"
Sep 17 17:55:59.058200 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:55:59.058157    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8" podUID="8831241f-9fea-415f-a060-8953a4cac1cd" containerName="model-chainer-raw-hpa-cb1fd" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:56:02.475139 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:02.475107    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/ovn-acl-logging/0.log"
Sep 17 17:56:02.475595 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:02.475575    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/ovn-acl-logging/0.log"
Sep 17 17:56:04.058332 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:04.058289    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8" podUID="8831241f-9fea-415f-a060-8953a4cac1cd" containerName="model-chainer-raw-hpa-cb1fd" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:56:09.058555 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:09.058514    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8" podUID="8831241f-9fea-415f-a060-8953a4cac1cd" containerName="model-chainer-raw-hpa-cb1fd" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:56:11.149514 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:11.149489    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8"
Sep 17 17:56:11.224150 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:11.224117    2576 generic.go:356] "Generic (PLEG): container finished" podID="8831241f-9fea-415f-a060-8953a4cac1cd" containerID="48117f33669576edf78db58e1fbab550091f494acb0a63b1d2326dfcadc4139b" exitCode=0
Sep 17 17:56:11.224322 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:11.224155    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8"
Sep 17 17:56:11.224322 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:11.224159    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8" event={"ID":"8831241f-9fea-415f-a060-8953a4cac1cd","Type":"ContainerDied","Data":"48117f33669576edf78db58e1fbab550091f494acb0a63b1d2326dfcadc4139b"}
Sep 17 17:56:11.224322 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:11.224285    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8" event={"ID":"8831241f-9fea-415f-a060-8953a4cac1cd","Type":"ContainerDied","Data":"e712a2e6a38cb48bfc539b9e8785a693d6d77dbf0128c895e38ef1689db875f8"}
Sep 17 17:56:11.224322 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:11.224314    2576 scope.go:117] "RemoveContainer" containerID="48117f33669576edf78db58e1fbab550091f494acb0a63b1d2326dfcadc4139b"
Sep 17 17:56:11.231944 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:11.231927    2576 scope.go:117] "RemoveContainer" containerID="48117f33669576edf78db58e1fbab550091f494acb0a63b1d2326dfcadc4139b"
Sep 17 17:56:11.232236 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:56:11.232211    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48117f33669576edf78db58e1fbab550091f494acb0a63b1d2326dfcadc4139b\": container with ID starting with 48117f33669576edf78db58e1fbab550091f494acb0a63b1d2326dfcadc4139b not found: ID does not exist" containerID="48117f33669576edf78db58e1fbab550091f494acb0a63b1d2326dfcadc4139b"
Sep 17 17:56:11.232335 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:11.232240    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48117f33669576edf78db58e1fbab550091f494acb0a63b1d2326dfcadc4139b"} err="failed to get container status \"48117f33669576edf78db58e1fbab550091f494acb0a63b1d2326dfcadc4139b\": rpc error: code = NotFound desc = could not find container \"48117f33669576edf78db58e1fbab550091f494acb0a63b1d2326dfcadc4139b\": container with ID starting with 48117f33669576edf78db58e1fbab550091f494acb0a63b1d2326dfcadc4139b not found: ID does not exist"
Sep 17 17:56:11.256524 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:11.256495    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8831241f-9fea-415f-a060-8953a4cac1cd-openshift-service-ca-bundle\") pod \"8831241f-9fea-415f-a060-8953a4cac1cd\" (UID: \"8831241f-9fea-415f-a060-8953a4cac1cd\") "
Sep 17 17:56:11.256677 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:11.256554    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8831241f-9fea-415f-a060-8953a4cac1cd-proxy-tls\") pod \"8831241f-9fea-415f-a060-8953a4cac1cd\" (UID: \"8831241f-9fea-415f-a060-8953a4cac1cd\") "
Sep 17 17:56:11.256869 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:11.256850    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8831241f-9fea-415f-a060-8953a4cac1cd-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "8831241f-9fea-415f-a060-8953a4cac1cd" (UID: "8831241f-9fea-415f-a060-8953a4cac1cd"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:11.258483 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:11.258465    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8831241f-9fea-415f-a060-8953a4cac1cd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8831241f-9fea-415f-a060-8953a4cac1cd" (UID: "8831241f-9fea-415f-a060-8953a4cac1cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:56:11.358088 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:11.358042    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8831241f-9fea-415f-a060-8953a4cac1cd-proxy-tls\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:56:11.358088 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:11.358085    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8831241f-9fea-415f-a060-8953a4cac1cd-openshift-service-ca-bundle\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:56:11.545782 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:11.545751    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8"]
Sep 17 17:56:11.550415 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:11.550391    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-cb1fd-88f49cd48-sflj8"]
Sep 17 17:56:12.537019 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:56:12.536989    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8831241f-9fea-415f-a060-8953a4cac1cd" path="/var/lib/kubelet/pods/8831241f-9fea-415f-a060-8953a4cac1cd/volumes"
Sep 17 17:57:05.693393 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.693349    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"]
Sep 17 17:57:05.693942 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.693800    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="120ac961-8617-458b-b043-04a937010915" containerName="storage-initializer"
Sep 17 17:57:05.693942 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.693819    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="120ac961-8617-458b-b043-04a937010915" containerName="storage-initializer"
Sep 17 17:57:05.693942 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.693834    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="storage-initializer"
Sep 17 17:57:05.693942 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.693843    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="storage-initializer"
Sep 17 17:57:05.693942 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.693854    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kserve-container"
Sep 17 17:57:05.693942 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.693862    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kserve-container"
Sep 17 17:57:05.693942 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.693875    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8831241f-9fea-415f-a060-8953a4cac1cd" containerName="model-chainer-raw-hpa-cb1fd"
Sep 17 17:57:05.693942 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.693884    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8831241f-9fea-415f-a060-8953a4cac1cd" containerName="model-chainer-raw-hpa-cb1fd"
Sep 17 17:57:05.693942 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.693896    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kube-rbac-proxy"
Sep 17 17:57:05.693942 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.693905    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kube-rbac-proxy"
Sep 17 17:57:05.693942 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.693917    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="120ac961-8617-458b-b043-04a937010915" containerName="kube-rbac-proxy"
Sep 17 17:57:05.693942 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.693925    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="120ac961-8617-458b-b043-04a937010915" containerName="kube-rbac-proxy"
Sep 17 17:57:05.693942 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.693937    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="120ac961-8617-458b-b043-04a937010915" containerName="kserve-container"
Sep 17 17:57:05.693942 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.693944    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="120ac961-8617-458b-b043-04a937010915" containerName="kserve-container"
Sep 17 17:57:05.694623 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.694030    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="120ac961-8617-458b-b043-04a937010915" containerName="kserve-container"
Sep 17 17:57:05.694623 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.694041    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8831241f-9fea-415f-a060-8953a4cac1cd" containerName="model-chainer-raw-hpa-cb1fd"
Sep 17 17:57:05.694623 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.694051    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kube-rbac-proxy"
Sep 17 17:57:05.694623 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.694059    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="63146f5e-47b2-4076-8ee7-6f3e34e181c2" containerName="kserve-container"
Sep 17 17:57:05.694623 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.694067    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="120ac961-8617-458b-b043-04a937010915" containerName="kube-rbac-proxy"
Sep 17 17:57:05.698777 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.698756    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"
Sep 17 17:57:05.701190 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.701168    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:57:05.701300 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.701244    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:57:05.702242 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.702219    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:57:05.702348 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.702256    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-5a8a9a-dockercfg-v9q49\""
Sep 17 17:57:05.702348 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.702274    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-5a8a9a-kube-rbac-proxy-sar-config\""
Sep 17 17:57:05.702459 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.702373    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-5a8a9a\""
Sep 17 17:57:05.702525 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.702515    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-5a8a9a-predictor-serving-cert\""
Sep 17 17:57:05.711976 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.711957    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"]
Sep 17 17:57:05.851547 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.851509    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6da9bf20-eaa5-47b9-bd66-37f162510117-cabundle-cert\") pod \"isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") " pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"
Sep 17 17:57:05.851750 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.851578    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6da9bf20-eaa5-47b9-bd66-37f162510117-kserve-provision-location\") pod \"isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") " pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"
Sep 17 17:57:05.851750 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.851611    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkqpp\" (UniqueName: \"kubernetes.io/projected/6da9bf20-eaa5-47b9-bd66-37f162510117-kube-api-access-vkqpp\") pod \"isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") " pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"
Sep 17 17:57:05.851750 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.851715    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6da9bf20-eaa5-47b9-bd66-37f162510117-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") " pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"
Sep 17 17:57:05.851869 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.851766    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6da9bf20-eaa5-47b9-bd66-37f162510117-proxy-tls\") pod \"isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") " pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"
Sep 17 17:57:05.952920 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.952828    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6da9bf20-eaa5-47b9-bd66-37f162510117-cabundle-cert\") pod \"isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") " pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"
Sep 17 17:57:05.952920 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.952870    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6da9bf20-eaa5-47b9-bd66-37f162510117-kserve-provision-location\") pod \"isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") " pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"
Sep 17 17:57:05.952920 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.952896    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkqpp\" (UniqueName: \"kubernetes.io/projected/6da9bf20-eaa5-47b9-bd66-37f162510117-kube-api-access-vkqpp\") pod \"isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") " pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"
Sep 17 17:57:05.953188 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.952944    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6da9bf20-eaa5-47b9-bd66-37f162510117-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") " pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"
Sep 17 17:57:05.953188 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.952989    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6da9bf20-eaa5-47b9-bd66-37f162510117-proxy-tls\") pod \"isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") " pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"
Sep 17 17:57:05.953383 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.953360    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6da9bf20-eaa5-47b9-bd66-37f162510117-kserve-provision-location\") pod \"isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") " pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"
Sep 17 17:57:05.953591 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.953529    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6da9bf20-eaa5-47b9-bd66-37f162510117-cabundle-cert\") pod \"isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") " pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"
Sep 17 17:57:05.953704 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.953631    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6da9bf20-eaa5-47b9-bd66-37f162510117-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") " pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"
Sep 17 17:57:05.955430 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.955408    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6da9bf20-eaa5-47b9-bd66-37f162510117-proxy-tls\") pod \"isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") " pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"
Sep 17 17:57:05.962191 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:05.962161    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkqpp\" (UniqueName: \"kubernetes.io/projected/6da9bf20-eaa5-47b9-bd66-37f162510117-kube-api-access-vkqpp\") pod \"isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") " pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"
Sep 17 17:57:06.009376 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:06.009342    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"
Sep 17 17:57:06.284624 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:06.284528    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"]
Sep 17 17:57:06.287592 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:57:06.287548    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da9bf20_eaa5_47b9_bd66_37f162510117.slice/crio-f91a24b81e299ff7dd88a28339e8b69099763cc988b2768e1061633e3ce7e80a WatchSource:0}: Error finding container f91a24b81e299ff7dd88a28339e8b69099763cc988b2768e1061633e3ce7e80a: Status 404 returned error can't find the container with id f91a24b81e299ff7dd88a28339e8b69099763cc988b2768e1061633e3ce7e80a
Sep 17 17:57:06.444657 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:06.444621    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8" event={"ID":"6da9bf20-eaa5-47b9-bd66-37f162510117","Type":"ContainerStarted","Data":"13e637ddb3494e3ded634d765ee29f3e1662d303f64d226a2fc8329236d55f6d"}
Sep 17 17:57:06.444846 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:06.444690    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8" event={"ID":"6da9bf20-eaa5-47b9-bd66-37f162510117","Type":"ContainerStarted","Data":"f91a24b81e299ff7dd88a28339e8b69099763cc988b2768e1061633e3ce7e80a"}
Sep 17 17:57:11.472493 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:11.472464    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8_6da9bf20-eaa5-47b9-bd66-37f162510117/storage-initializer/0.log"
Sep 17 17:57:11.472888 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:11.472501    2576 generic.go:356] "Generic (PLEG): container finished" podID="6da9bf20-eaa5-47b9-bd66-37f162510117" containerID="13e637ddb3494e3ded634d765ee29f3e1662d303f64d226a2fc8329236d55f6d" exitCode=1
Sep 17 17:57:11.472888 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:11.472584    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8" event={"ID":"6da9bf20-eaa5-47b9-bd66-37f162510117","Type":"ContainerDied","Data":"13e637ddb3494e3ded634d765ee29f3e1662d303f64d226a2fc8329236d55f6d"}
Sep 17 17:57:12.478799 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:12.478771    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8_6da9bf20-eaa5-47b9-bd66-37f162510117/storage-initializer/0.log"
Sep 17 17:57:12.479210 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:12.478854    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8" event={"ID":"6da9bf20-eaa5-47b9-bd66-37f162510117","Type":"ContainerStarted","Data":"463178f057345477e99e3d521fdbb7d62e1698731555591b1e8958b1dbb1e484"}
Sep 17 17:57:18.504928 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:18.504899    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8_6da9bf20-eaa5-47b9-bd66-37f162510117/storage-initializer/1.log"
Sep 17 17:57:18.505381 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:18.505336    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8_6da9bf20-eaa5-47b9-bd66-37f162510117/storage-initializer/0.log"
Sep 17 17:57:18.505447 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:18.505376    2576 generic.go:356] "Generic (PLEG): container finished" podID="6da9bf20-eaa5-47b9-bd66-37f162510117" containerID="463178f057345477e99e3d521fdbb7d62e1698731555591b1e8958b1dbb1e484" exitCode=1
Sep 17 17:57:18.505447 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:18.505411    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8" event={"ID":"6da9bf20-eaa5-47b9-bd66-37f162510117","Type":"ContainerDied","Data":"463178f057345477e99e3d521fdbb7d62e1698731555591b1e8958b1dbb1e484"}
Sep 17 17:57:18.505553 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:18.505485    2576 scope.go:117] "RemoveContainer" containerID="13e637ddb3494e3ded634d765ee29f3e1662d303f64d226a2fc8329236d55f6d"
Sep 17 17:57:18.505911 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:18.505885    2576 scope.go:117] "RemoveContainer" containerID="13e637ddb3494e3ded634d765ee29f3e1662d303f64d226a2fc8329236d55f6d"
Sep 17 17:57:18.516208 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:57:18.516180    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8_kserve-ci-e2e-test(6da9bf20-eaa5-47b9-bd66-37f162510117)\"" pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8" podUID="6da9bf20-eaa5-47b9-bd66-37f162510117"
Sep 17 17:57:18.516466 ip-10-0-136-127 kubenswrapper[2576]: E0917 17:57:18.516437    2576 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8_kserve-ci-e2e-test_6da9bf20-eaa5-47b9-bd66-37f162510117_0 in pod sandbox f91a24b81e299ff7dd88a28339e8b69099763cc988b2768e1061633e3ce7e80a from index: no such id: '13e637ddb3494e3ded634d765ee29f3e1662d303f64d226a2fc8329236d55f6d'" containerID="13e637ddb3494e3ded634d765ee29f3e1662d303f64d226a2fc8329236d55f6d"
Sep 17 17:57:18.516530 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:18.516482    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e637ddb3494e3ded634d765ee29f3e1662d303f64d226a2fc8329236d55f6d"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8_kserve-ci-e2e-test_6da9bf20-eaa5-47b9-bd66-37f162510117_0 in pod sandbox f91a24b81e299ff7dd88a28339e8b69099763cc988b2768e1061633e3ce7e80a from index: no such id: '13e637ddb3494e3ded634d765ee29f3e1662d303f64d226a2fc8329236d55f6d'"
Sep 17 17:57:19.511423 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:19.511395    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8_6da9bf20-eaa5-47b9-bd66-37f162510117/storage-initializer/1.log"
Sep 17 17:57:21.780766 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:21.780733    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"]
Sep 17 17:57:21.910653 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:21.910629    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8_6da9bf20-eaa5-47b9-bd66-37f162510117/storage-initializer/1.log"
Sep 17 17:57:21.912790 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:21.912774    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"
Sep 17 17:57:21.972809 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:21.972769    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6da9bf20-eaa5-47b9-bd66-37f162510117-cabundle-cert\") pod \"6da9bf20-eaa5-47b9-bd66-37f162510117\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") "
Sep 17 17:57:21.973004 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:21.972828    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkqpp\" (UniqueName: \"kubernetes.io/projected/6da9bf20-eaa5-47b9-bd66-37f162510117-kube-api-access-vkqpp\") pod \"6da9bf20-eaa5-47b9-bd66-37f162510117\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") "
Sep 17 17:57:21.973004 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:21.972879    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6da9bf20-eaa5-47b9-bd66-37f162510117-proxy-tls\") pod \"6da9bf20-eaa5-47b9-bd66-37f162510117\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") "
Sep 17 17:57:21.973004 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:21.972907    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6da9bf20-eaa5-47b9-bd66-37f162510117-kserve-provision-location\") pod \"6da9bf20-eaa5-47b9-bd66-37f162510117\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") "
Sep 17 17:57:21.973004 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:21.972942    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6da9bf20-eaa5-47b9-bd66-37f162510117-kube-rbac-proxy-sar-config\") pod \"6da9bf20-eaa5-47b9-bd66-37f162510117\" (UID: \"6da9bf20-eaa5-47b9-bd66-37f162510117\") "
Sep 17 17:57:21.973195 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:21.973134    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da9bf20-eaa5-47b9-bd66-37f162510117-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6da9bf20-eaa5-47b9-bd66-37f162510117" (UID: "6da9bf20-eaa5-47b9-bd66-37f162510117"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:21.973195 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:21.973175    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da9bf20-eaa5-47b9-bd66-37f162510117-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6da9bf20-eaa5-47b9-bd66-37f162510117" (UID: "6da9bf20-eaa5-47b9-bd66-37f162510117"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:57:21.973330 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:21.973310    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da9bf20-eaa5-47b9-bd66-37f162510117-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6da9bf20-eaa5-47b9-bd66-37f162510117" (UID: "6da9bf20-eaa5-47b9-bd66-37f162510117"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:57:21.974978 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:21.974956    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da9bf20-eaa5-47b9-bd66-37f162510117-kube-api-access-vkqpp" (OuterVolumeSpecName: "kube-api-access-vkqpp") pod "6da9bf20-eaa5-47b9-bd66-37f162510117" (UID: "6da9bf20-eaa5-47b9-bd66-37f162510117"). InnerVolumeSpecName "kube-api-access-vkqpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:57:21.975062 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:21.974985    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6da9bf20-eaa5-47b9-bd66-37f162510117-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6da9bf20-eaa5-47b9-bd66-37f162510117" (UID: "6da9bf20-eaa5-47b9-bd66-37f162510117"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:57:22.073678 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:22.073551    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6da9bf20-eaa5-47b9-bd66-37f162510117-proxy-tls\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:57:22.073678 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:22.073623    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6da9bf20-eaa5-47b9-bd66-37f162510117-kserve-provision-location\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:57:22.073678 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:22.073635    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6da9bf20-eaa5-47b9-bd66-37f162510117-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:57:22.073678 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:22.073645    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6da9bf20-eaa5-47b9-bd66-37f162510117-cabundle-cert\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:57:22.073678 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:22.073653    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkqpp\" (UniqueName: \"kubernetes.io/projected/6da9bf20-eaa5-47b9-bd66-37f162510117-kube-api-access-vkqpp\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 17:57:22.525919 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:22.525890    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8_6da9bf20-eaa5-47b9-bd66-37f162510117/storage-initializer/1.log"
Sep 17 17:57:22.526085 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:22.525972    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8" event={"ID":"6da9bf20-eaa5-47b9-bd66-37f162510117","Type":"ContainerDied","Data":"f91a24b81e299ff7dd88a28339e8b69099763cc988b2768e1061633e3ce7e80a"}
Sep 17 17:57:22.526085 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:22.526006    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"
Sep 17 17:57:22.526085 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:22.526053    2576 scope.go:117] "RemoveContainer" containerID="463178f057345477e99e3d521fdbb7d62e1698731555591b1e8958b1dbb1e484"
Sep 17 17:57:22.566366 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:22.566336    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"]
Sep 17 17:57:22.571382 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:22.571357    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5a8a9a-predictor-6cd7688dc-qmwx8"]
Sep 17 17:57:24.537713 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:24.537677    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6da9bf20-eaa5-47b9-bd66-37f162510117" path="/var/lib/kubelet/pods/6da9bf20-eaa5-47b9-bd66-37f162510117/volumes"
Sep 17 17:57:26.105618 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.105587    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"]
Sep 17 17:57:26.106036 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.105923    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6da9bf20-eaa5-47b9-bd66-37f162510117" containerName="storage-initializer"
Sep 17 17:57:26.106036 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.105936    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6da9bf20-eaa5-47b9-bd66-37f162510117" containerName="storage-initializer"
Sep 17 17:57:26.106036 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.105946    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6da9bf20-eaa5-47b9-bd66-37f162510117" containerName="storage-initializer"
Sep 17 17:57:26.106036 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.105952    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6da9bf20-eaa5-47b9-bd66-37f162510117" containerName="storage-initializer"
Sep 17 17:57:26.106036 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.106007    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6da9bf20-eaa5-47b9-bd66-37f162510117" containerName="storage-initializer"
Sep 17 17:57:26.106195 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.106103    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6da9bf20-eaa5-47b9-bd66-37f162510117" containerName="storage-initializer"
Sep 17 17:57:26.111956 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.111923    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:26.115888 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.115863    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-92dc8-kube-rbac-proxy-sar-config\""
Sep 17 17:57:26.116747 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.116721    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-92dc8-predictor-serving-cert\""
Sep 17 17:57:26.117070 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.117050    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:57:26.117166 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.117070    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:57:26.118550 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.118531    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gwm88\""
Sep 17 17:57:26.118550 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.118540    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:57:26.118745 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.118614    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:57:26.122808 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.122784    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"]
Sep 17 17:57:26.202728 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.202693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ad32257-4301-4614-91e0-542b9652e6af-cabundle-cert\") pod \"isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:26.202922 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.202742    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ad32257-4301-4614-91e0-542b9652e6af-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:26.202922 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.202768    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ad32257-4301-4614-91e0-542b9652e6af-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:26.202922 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.202802    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ad32257-4301-4614-91e0-542b9652e6af-proxy-tls\") pod \"isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:26.202922 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.202892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt45l\" (UniqueName: \"kubernetes.io/projected/1ad32257-4301-4614-91e0-542b9652e6af-kube-api-access-pt45l\") pod \"isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:26.303541 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.303502    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ad32257-4301-4614-91e0-542b9652e6af-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:26.303541 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.303544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ad32257-4301-4614-91e0-542b9652e6af-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:26.303875 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.303607    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ad32257-4301-4614-91e0-542b9652e6af-proxy-tls\") pod \"isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:26.303875 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.303673    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt45l\" (UniqueName: \"kubernetes.io/projected/1ad32257-4301-4614-91e0-542b9652e6af-kube-api-access-pt45l\") pod \"isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:26.303875 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.303707    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ad32257-4301-4614-91e0-542b9652e6af-cabundle-cert\") pod \"isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:26.304090 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.304068    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ad32257-4301-4614-91e0-542b9652e6af-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:26.304304 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.304279    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ad32257-4301-4614-91e0-542b9652e6af-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:26.304369 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.304305    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ad32257-4301-4614-91e0-542b9652e6af-cabundle-cert\") pod \"isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:26.306106 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.306087    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ad32257-4301-4614-91e0-542b9652e6af-proxy-tls\") pod \"isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:26.312357 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.312327    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt45l\" (UniqueName: \"kubernetes.io/projected/1ad32257-4301-4614-91e0-542b9652e6af-kube-api-access-pt45l\") pod \"isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:26.424882 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.424783    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:26.711888 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:26.711803    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"]
Sep 17 17:57:26.714906 ip-10-0-136-127 kubenswrapper[2576]: W0917 17:57:26.714876    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad32257_4301_4614_91e0_542b9652e6af.slice/crio-1c4497b21fe5cdf501e8b6c924609439d14fd4a927e495ea88bf62aa1c0ab66f WatchSource:0}: Error finding container 1c4497b21fe5cdf501e8b6c924609439d14fd4a927e495ea88bf62aa1c0ab66f: Status 404 returned error can't find the container with id 1c4497b21fe5cdf501e8b6c924609439d14fd4a927e495ea88bf62aa1c0ab66f
Sep 17 17:57:27.548873 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:27.548838    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" event={"ID":"1ad32257-4301-4614-91e0-542b9652e6af","Type":"ContainerStarted","Data":"5eaa2a26ffc31cd4576e97ed33c74c8fb0617b534d7ffb4a2adec939a01a398d"}
Sep 17 17:57:27.549271 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:27.548908    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" event={"ID":"1ad32257-4301-4614-91e0-542b9652e6af","Type":"ContainerStarted","Data":"1c4497b21fe5cdf501e8b6c924609439d14fd4a927e495ea88bf62aa1c0ab66f"}
Sep 17 17:57:31.572456 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:31.572422    2576 generic.go:356] "Generic (PLEG): container finished" podID="1ad32257-4301-4614-91e0-542b9652e6af" containerID="5eaa2a26ffc31cd4576e97ed33c74c8fb0617b534d7ffb4a2adec939a01a398d" exitCode=0
Sep 17 17:57:31.572901 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:31.572490    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" event={"ID":"1ad32257-4301-4614-91e0-542b9652e6af","Type":"ContainerDied","Data":"5eaa2a26ffc31cd4576e97ed33c74c8fb0617b534d7ffb4a2adec939a01a398d"}
Sep 17 17:57:32.578362 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:32.578328    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" event={"ID":"1ad32257-4301-4614-91e0-542b9652e6af","Type":"ContainerStarted","Data":"174abb03eed86c31a4c2d474ad351dee21503e7f0c2069cc92fe2a3b80a635c1"}
Sep 17 17:57:32.578764 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:32.578386    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" event={"ID":"1ad32257-4301-4614-91e0-542b9652e6af","Type":"ContainerStarted","Data":"cb2aa1fb5a6e27fb5ae88098ddae97e8cb404017d247d66f82babaa091d434f3"}
Sep 17 17:57:32.578764 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:32.578671    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:32.578764 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:32.578715    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:32.580062 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:32.580033    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:57:32.601465 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:32.601422    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podStartSLOduration=2.826415656 podStartE2EDuration="6.601409281s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.774993625s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:32.600089928 +0000 UTC m=+992.192926911" watchObservedRunningTime="2026-09-17 17:57:32.601409281 +0000 UTC m=+992.194246263"
Sep 17 17:57:33.583024 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:33.582983    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:57:38.587361 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:38.587327    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 17:57:38.587838 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:38.587778    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:57:48.587833 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:48.587786    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:57:58.588459 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:57:58.588419    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:58:08.588641 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:58:08.588600    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:58:18.588413 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:58:18.588371    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:58:28.588422 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:58:28.588385    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:58:38.588550 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:58:38.588507    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:58:48.588779 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:58:48.588738    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:58:58.588764 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:58:58.588723    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:59:08.533262 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:59:08.533217    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:59:18.533127 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:59:18.533090    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:59:28.533892 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:59:28.533842    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:59:38.533087 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:59:38.533046    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:59:48.533380 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:59:48.533337    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 17:59:58.533745 ip-10-0-136-127 kubenswrapper[2576]: I0917 17:59:58.533700    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 18:00:08.537613 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:08.537585    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 18:00:16.108806 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:16.108770    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"]
Sep 17 18:00:16.109305 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:16.109082    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container" containerID="cri-o://cb2aa1fb5a6e27fb5ae88098ddae97e8cb404017d247d66f82babaa091d434f3" gracePeriod=30
Sep 17 18:00:16.109305 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:16.109128    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kube-rbac-proxy" containerID="cri-o://174abb03eed86c31a4c2d474ad351dee21503e7f0c2069cc92fe2a3b80a635c1" gracePeriod=30
Sep 17 18:00:16.251091 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:16.251056    2576 generic.go:356] "Generic (PLEG): container finished" podID="1ad32257-4301-4614-91e0-542b9652e6af" containerID="174abb03eed86c31a4c2d474ad351dee21503e7f0c2069cc92fe2a3b80a635c1" exitCode=2
Sep 17 18:00:16.251208 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:16.251105    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" event={"ID":"1ad32257-4301-4614-91e0-542b9652e6af","Type":"ContainerDied","Data":"174abb03eed86c31a4c2d474ad351dee21503e7f0c2069cc92fe2a3b80a635c1"}
Sep 17 18:00:18.533813 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:18.533766    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 18:00:18.584021 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:18.583988    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 17 18:00:21.444922 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:21.444884    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"]
Sep 17 18:00:21.448250 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:21.448227    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:21.451110 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:21.451087    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 18:00:21.451599 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:21.451585    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 18:00:21.462549 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:21.462526    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"]
Sep 17 18:00:21.566485 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:21.566462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52920a25-a151-4742-ba02-abc02af29a3b-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-pttm5\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:21.566615 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:21.566494    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52920a25-a151-4742-ba02-abc02af29a3b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-pttm5\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:21.566615 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:21.566512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52920a25-a151-4742-ba02-abc02af29a3b-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-pttm5\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:21.566615 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:21.566531    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkzcs\" (UniqueName: \"kubernetes.io/projected/52920a25-a151-4742-ba02-abc02af29a3b-kube-api-access-gkzcs\") pod \"isvc-canary-create-predictor-5f845bd94d-pttm5\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:21.566733 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:21.566657    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52920a25-a151-4742-ba02-abc02af29a3b-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-pttm5\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:21.667960 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:21.667921    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52920a25-a151-4742-ba02-abc02af29a3b-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-pttm5\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:21.668114 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:21.667979    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52920a25-a151-4742-ba02-abc02af29a3b-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-pttm5\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:21.668114 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:21.668005    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52920a25-a151-4742-ba02-abc02af29a3b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-pttm5\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:21.668114 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:21.668022    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52920a25-a151-4742-ba02-abc02af29a3b-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-pttm5\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:21.668114 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:21.668040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkzcs\" (UniqueName: \"kubernetes.io/projected/52920a25-a151-4742-ba02-abc02af29a3b-kube-api-access-gkzcs\") pod \"isvc-canary-create-predictor-5f845bd94d-pttm5\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:21.668333 ip-10-0-136-127 kubenswrapper[2576]: E0917 18:00:21.668201    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 18:00:21.668333 ip-10-0-136-127 kubenswrapper[2576]: E0917 18:00:21.668298    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52920a25-a151-4742-ba02-abc02af29a3b-proxy-tls podName:52920a25-a151-4742-ba02-abc02af29a3b nodeName:}" failed. No retries permitted until 2026-09-17 18:00:22.168279232 +0000 UTC m=+1161.761116192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/52920a25-a151-4742-ba02-abc02af29a3b-proxy-tls") pod "isvc-canary-create-predictor-5f845bd94d-pttm5" (UID: "52920a25-a151-4742-ba02-abc02af29a3b") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 18:00:21.668448 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:21.668337    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52920a25-a151-4742-ba02-abc02af29a3b-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f845bd94d-pttm5\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:21.668748 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:21.668725    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52920a25-a151-4742-ba02-abc02af29a3b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f845bd94d-pttm5\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:21.668828 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:21.668754    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52920a25-a151-4742-ba02-abc02af29a3b-cabundle-cert\") pod \"isvc-canary-create-predictor-5f845bd94d-pttm5\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:21.679107 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:21.679083    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkzcs\" (UniqueName: \"kubernetes.io/projected/52920a25-a151-4742-ba02-abc02af29a3b-kube-api-access-gkzcs\") pod \"isvc-canary-create-predictor-5f845bd94d-pttm5\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:22.171701 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:22.171672    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52920a25-a151-4742-ba02-abc02af29a3b-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-pttm5\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:22.174045 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:22.174019    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52920a25-a151-4742-ba02-abc02af29a3b-proxy-tls\") pod \"isvc-canary-create-predictor-5f845bd94d-pttm5\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:22.359016 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:22.358988    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:22.648017 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:22.647996    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"]
Sep 17 18:00:22.650512 ip-10-0-136-127 kubenswrapper[2576]: W0917 18:00:22.650484    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52920a25_a151_4742_ba02_abc02af29a3b.slice/crio-862ac86a6b95504cce8c5234ee6bc3b9b924cabc7c978e8c680b2a005d855715 WatchSource:0}: Error finding container 862ac86a6b95504cce8c5234ee6bc3b9b924cabc7c978e8c680b2a005d855715: Status 404 returned error can't find the container with id 862ac86a6b95504cce8c5234ee6bc3b9b924cabc7c978e8c680b2a005d855715
Sep 17 18:00:23.280188 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:23.280150    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" event={"ID":"52920a25-a151-4742-ba02-abc02af29a3b","Type":"ContainerStarted","Data":"d2adba548bca039fc010bf95833c36c7eeb76392912f348b405e5e7c3ed9eb72"}
Sep 17 18:00:23.280381 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:23.280223    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" event={"ID":"52920a25-a151-4742-ba02-abc02af29a3b","Type":"ContainerStarted","Data":"862ac86a6b95504cce8c5234ee6bc3b9b924cabc7c978e8c680b2a005d855715"}
Sep 17 18:00:23.583702 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:23.583665    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 17 18:00:25.291949 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:25.291839    2576 generic.go:356] "Generic (PLEG): container finished" podID="1ad32257-4301-4614-91e0-542b9652e6af" containerID="cb2aa1fb5a6e27fb5ae88098ddae97e8cb404017d247d66f82babaa091d434f3" exitCode=0
Sep 17 18:00:25.291949 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:25.291902    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" event={"ID":"1ad32257-4301-4614-91e0-542b9652e6af","Type":"ContainerDied","Data":"cb2aa1fb5a6e27fb5ae88098ddae97e8cb404017d247d66f82babaa091d434f3"}
Sep 17 18:00:25.358060 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:25.358037    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 18:00:25.496749 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:25.496669    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ad32257-4301-4614-91e0-542b9652e6af-kserve-provision-location\") pod \"1ad32257-4301-4614-91e0-542b9652e6af\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") "
Sep 17 18:00:25.496749 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:25.496714    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ad32257-4301-4614-91e0-542b9652e6af-kube-rbac-proxy-sar-config\") pod \"1ad32257-4301-4614-91e0-542b9652e6af\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") "
Sep 17 18:00:25.496749 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:25.496733    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pt45l\" (UniqueName: \"kubernetes.io/projected/1ad32257-4301-4614-91e0-542b9652e6af-kube-api-access-pt45l\") pod \"1ad32257-4301-4614-91e0-542b9652e6af\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") "
Sep 17 18:00:25.496749 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:25.496753    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ad32257-4301-4614-91e0-542b9652e6af-proxy-tls\") pod \"1ad32257-4301-4614-91e0-542b9652e6af\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") "
Sep 17 18:00:25.497074 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:25.496771    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ad32257-4301-4614-91e0-542b9652e6af-cabundle-cert\") pod \"1ad32257-4301-4614-91e0-542b9652e6af\" (UID: \"1ad32257-4301-4614-91e0-542b9652e6af\") "
Sep 17 18:00:25.497074 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:25.497016    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad32257-4301-4614-91e0-542b9652e6af-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ad32257-4301-4614-91e0-542b9652e6af" (UID: "1ad32257-4301-4614-91e0-542b9652e6af"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:25.497170 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:25.497087    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad32257-4301-4614-91e0-542b9652e6af-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1ad32257-4301-4614-91e0-542b9652e6af" (UID: "1ad32257-4301-4614-91e0-542b9652e6af"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:25.497170 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:25.497145    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad32257-4301-4614-91e0-542b9652e6af-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1ad32257-4301-4614-91e0-542b9652e6af" (UID: "1ad32257-4301-4614-91e0-542b9652e6af"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:25.498894 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:25.498865    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad32257-4301-4614-91e0-542b9652e6af-kube-api-access-pt45l" (OuterVolumeSpecName: "kube-api-access-pt45l") pod "1ad32257-4301-4614-91e0-542b9652e6af" (UID: "1ad32257-4301-4614-91e0-542b9652e6af"). InnerVolumeSpecName "kube-api-access-pt45l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:25.498994 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:25.498896    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ad32257-4301-4614-91e0-542b9652e6af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ad32257-4301-4614-91e0-542b9652e6af" (UID: "1ad32257-4301-4614-91e0-542b9652e6af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:25.598063 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:25.598025    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ad32257-4301-4614-91e0-542b9652e6af-proxy-tls\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:00:25.598063 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:25.598055    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ad32257-4301-4614-91e0-542b9652e6af-cabundle-cert\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:00:25.598063 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:25.598064    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ad32257-4301-4614-91e0-542b9652e6af-kserve-provision-location\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:00:25.598063 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:25.598074    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ad32257-4301-4614-91e0-542b9652e6af-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:00:25.598320 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:25.598083    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt45l\" (UniqueName: \"kubernetes.io/projected/1ad32257-4301-4614-91e0-542b9652e6af-kube-api-access-pt45l\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:00:26.299020 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:26.298982    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr" event={"ID":"1ad32257-4301-4614-91e0-542b9652e6af","Type":"ContainerDied","Data":"1c4497b21fe5cdf501e8b6c924609439d14fd4a927e495ea88bf62aa1c0ab66f"}
Sep 17 18:00:26.299020 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:26.299010    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"
Sep 17 18:00:26.299511 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:26.299061    2576 scope.go:117] "RemoveContainer" containerID="174abb03eed86c31a4c2d474ad351dee21503e7f0c2069cc92fe2a3b80a635c1"
Sep 17 18:00:26.307807 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:26.307792    2576 scope.go:117] "RemoveContainer" containerID="cb2aa1fb5a6e27fb5ae88098ddae97e8cb404017d247d66f82babaa091d434f3"
Sep 17 18:00:26.314915 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:26.314895    2576 scope.go:117] "RemoveContainer" containerID="5eaa2a26ffc31cd4576e97ed33c74c8fb0617b534d7ffb4a2adec939a01a398d"
Sep 17 18:00:26.322956 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:26.322938    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"]
Sep 17 18:00:26.325981 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:26.325962    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-92dc8-predictor-697694b787-5dvkr"]
Sep 17 18:00:26.538361 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:26.538327    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ad32257-4301-4614-91e0-542b9652e6af" path="/var/lib/kubelet/pods/1ad32257-4301-4614-91e0-542b9652e6af/volumes"
Sep 17 18:00:27.305847 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:27.305816    2576 generic.go:356] "Generic (PLEG): container finished" podID="52920a25-a151-4742-ba02-abc02af29a3b" containerID="d2adba548bca039fc010bf95833c36c7eeb76392912f348b405e5e7c3ed9eb72" exitCode=0
Sep 17 18:00:27.306249 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:27.305868    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" event={"ID":"52920a25-a151-4742-ba02-abc02af29a3b","Type":"ContainerDied","Data":"d2adba548bca039fc010bf95833c36c7eeb76392912f348b405e5e7c3ed9eb72"}
Sep 17 18:00:28.312579 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:28.312532    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" event={"ID":"52920a25-a151-4742-ba02-abc02af29a3b","Type":"ContainerStarted","Data":"077375e7f6bae35c573f424989688ca7ae73dc1c5c18eb99eb30e1bd669d7925"}
Sep 17 18:00:28.313031 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:28.312630    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" event={"ID":"52920a25-a151-4742-ba02-abc02af29a3b","Type":"ContainerStarted","Data":"7ed1da68c1c0f52789ab0f36eb2b270c877a69da11f3be1b4196ff4eac134c94"}
Sep 17 18:00:28.313031 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:28.312859    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:28.334485 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:28.334387    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" podStartSLOduration=3.369481002 podStartE2EDuration="7.334369631s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.964888629s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:28.333894215 +0000 UTC m=+1167.926731218" watchObservedRunningTime="2026-09-17 18:00:28.334369631 +0000 UTC m=+1167.927206614"
Sep 17 18:00:29.316514 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:29.316485    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:29.317857 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:29.317828    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:00:30.320101 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:30.320065    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:00:35.325637 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:35.325601    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:00:35.326141 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:35.326118    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:00:45.326323 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:45.326283    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:00:55.326230 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:00:55.326189    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:01:02.499415 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:02.499384    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/ovn-acl-logging/0.log"
Sep 17 18:01:02.500932 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:02.500915    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/ovn-acl-logging/0.log"
Sep 17 18:01:05.326724 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:05.326687    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:01:15.326363 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:15.326326    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:01:25.326147 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:25.326101    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:01:35.326264 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:35.326227    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:01:44.592873 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:44.592779    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"]
Sep 17 18:01:44.593330 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:44.593078    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="kserve-container" containerID="cri-o://7ed1da68c1c0f52789ab0f36eb2b270c877a69da11f3be1b4196ff4eac134c94" gracePeriod=30
Sep 17 18:01:44.593330 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:44.593152    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="kube-rbac-proxy" containerID="cri-o://077375e7f6bae35c573f424989688ca7ae73dc1c5c18eb99eb30e1bd669d7925" gracePeriod=30
Sep 17 18:01:44.594057 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:44.593979    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:01:45.321162 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:45.321121    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 18:01:45.616240 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:45.616209    2576 generic.go:356] "Generic (PLEG): container finished" podID="52920a25-a151-4742-ba02-abc02af29a3b" containerID="077375e7f6bae35c573f424989688ca7ae73dc1c5c18eb99eb30e1bd669d7925" exitCode=2
Sep 17 18:01:45.616616 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:45.616265    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" event={"ID":"52920a25-a151-4742-ba02-abc02af29a3b","Type":"ContainerDied","Data":"077375e7f6bae35c573f424989688ca7ae73dc1c5c18eb99eb30e1bd669d7925"}
Sep 17 18:01:48.939768 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:48.939744    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:01:49.034311 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.034250    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52920a25-a151-4742-ba02-abc02af29a3b-cabundle-cert\") pod \"52920a25-a151-4742-ba02-abc02af29a3b\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") "
Sep 17 18:01:49.034311 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.034290    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52920a25-a151-4742-ba02-abc02af29a3b-kserve-provision-location\") pod \"52920a25-a151-4742-ba02-abc02af29a3b\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") "
Sep 17 18:01:49.034311 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.034310    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52920a25-a151-4742-ba02-abc02af29a3b-kube-rbac-proxy-sar-config\") pod \"52920a25-a151-4742-ba02-abc02af29a3b\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") "
Sep 17 18:01:49.034511 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.034366    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52920a25-a151-4742-ba02-abc02af29a3b-proxy-tls\") pod \"52920a25-a151-4742-ba02-abc02af29a3b\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") "
Sep 17 18:01:49.034511 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.034383    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkzcs\" (UniqueName: \"kubernetes.io/projected/52920a25-a151-4742-ba02-abc02af29a3b-kube-api-access-gkzcs\") pod \"52920a25-a151-4742-ba02-abc02af29a3b\" (UID: \"52920a25-a151-4742-ba02-abc02af29a3b\") "
Sep 17 18:01:49.034685 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.034658    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52920a25-a151-4742-ba02-abc02af29a3b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "52920a25-a151-4742-ba02-abc02af29a3b" (UID: "52920a25-a151-4742-ba02-abc02af29a3b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:49.034736 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.034658    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52920a25-a151-4742-ba02-abc02af29a3b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "52920a25-a151-4742-ba02-abc02af29a3b" (UID: "52920a25-a151-4742-ba02-abc02af29a3b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:49.034781 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.034760    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52920a25-a151-4742-ba02-abc02af29a3b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "52920a25-a151-4742-ba02-abc02af29a3b" (UID: "52920a25-a151-4742-ba02-abc02af29a3b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:49.036426 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.036403    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52920a25-a151-4742-ba02-abc02af29a3b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "52920a25-a151-4742-ba02-abc02af29a3b" (UID: "52920a25-a151-4742-ba02-abc02af29a3b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:01:49.036497 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.036478    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52920a25-a151-4742-ba02-abc02af29a3b-kube-api-access-gkzcs" (OuterVolumeSpecName: "kube-api-access-gkzcs") pod "52920a25-a151-4742-ba02-abc02af29a3b" (UID: "52920a25-a151-4742-ba02-abc02af29a3b"). InnerVolumeSpecName "kube-api-access-gkzcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:01:49.134946 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.134917    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52920a25-a151-4742-ba02-abc02af29a3b-kserve-provision-location\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:01:49.134946 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.134943    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52920a25-a151-4742-ba02-abc02af29a3b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:01:49.135109 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.134958    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52920a25-a151-4742-ba02-abc02af29a3b-proxy-tls\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:01:49.135109 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.134970    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkzcs\" (UniqueName: \"kubernetes.io/projected/52920a25-a151-4742-ba02-abc02af29a3b-kube-api-access-gkzcs\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:01:49.135109 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.134981    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52920a25-a151-4742-ba02-abc02af29a3b-cabundle-cert\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:01:49.634939 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.634903    2576 generic.go:356] "Generic (PLEG): container finished" podID="52920a25-a151-4742-ba02-abc02af29a3b" containerID="7ed1da68c1c0f52789ab0f36eb2b270c877a69da11f3be1b4196ff4eac134c94" exitCode=0
Sep 17 18:01:49.635129 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.634960    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"
Sep 17 18:01:49.635129 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.634977    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" event={"ID":"52920a25-a151-4742-ba02-abc02af29a3b","Type":"ContainerDied","Data":"7ed1da68c1c0f52789ab0f36eb2b270c877a69da11f3be1b4196ff4eac134c94"}
Sep 17 18:01:49.635129 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.635026    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5" event={"ID":"52920a25-a151-4742-ba02-abc02af29a3b","Type":"ContainerDied","Data":"862ac86a6b95504cce8c5234ee6bc3b9b924cabc7c978e8c680b2a005d855715"}
Sep 17 18:01:49.635129 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.635052    2576 scope.go:117] "RemoveContainer" containerID="077375e7f6bae35c573f424989688ca7ae73dc1c5c18eb99eb30e1bd669d7925"
Sep 17 18:01:49.643508 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.643479    2576 scope.go:117] "RemoveContainer" containerID="7ed1da68c1c0f52789ab0f36eb2b270c877a69da11f3be1b4196ff4eac134c94"
Sep 17 18:01:49.650608 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.650591    2576 scope.go:117] "RemoveContainer" containerID="d2adba548bca039fc010bf95833c36c7eeb76392912f348b405e5e7c3ed9eb72"
Sep 17 18:01:49.657763 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.657747    2576 scope.go:117] "RemoveContainer" containerID="077375e7f6bae35c573f424989688ca7ae73dc1c5c18eb99eb30e1bd669d7925"
Sep 17 18:01:49.658055 ip-10-0-136-127 kubenswrapper[2576]: E0917 18:01:49.658036    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077375e7f6bae35c573f424989688ca7ae73dc1c5c18eb99eb30e1bd669d7925\": container with ID starting with 077375e7f6bae35c573f424989688ca7ae73dc1c5c18eb99eb30e1bd669d7925 not found: ID does not exist" containerID="077375e7f6bae35c573f424989688ca7ae73dc1c5c18eb99eb30e1bd669d7925"
Sep 17 18:01:49.658128 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.658067    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077375e7f6bae35c573f424989688ca7ae73dc1c5c18eb99eb30e1bd669d7925"} err="failed to get container status \"077375e7f6bae35c573f424989688ca7ae73dc1c5c18eb99eb30e1bd669d7925\": rpc error: code = NotFound desc = could not find container \"077375e7f6bae35c573f424989688ca7ae73dc1c5c18eb99eb30e1bd669d7925\": container with ID starting with 077375e7f6bae35c573f424989688ca7ae73dc1c5c18eb99eb30e1bd669d7925 not found: ID does not exist"
Sep 17 18:01:49.658128 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.658090    2576 scope.go:117] "RemoveContainer" containerID="7ed1da68c1c0f52789ab0f36eb2b270c877a69da11f3be1b4196ff4eac134c94"
Sep 17 18:01:49.658355 ip-10-0-136-127 kubenswrapper[2576]: E0917 18:01:49.658338    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ed1da68c1c0f52789ab0f36eb2b270c877a69da11f3be1b4196ff4eac134c94\": container with ID starting with 7ed1da68c1c0f52789ab0f36eb2b270c877a69da11f3be1b4196ff4eac134c94 not found: ID does not exist" containerID="7ed1da68c1c0f52789ab0f36eb2b270c877a69da11f3be1b4196ff4eac134c94"
Sep 17 18:01:49.658390 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.658361    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed1da68c1c0f52789ab0f36eb2b270c877a69da11f3be1b4196ff4eac134c94"} err="failed to get container status \"7ed1da68c1c0f52789ab0f36eb2b270c877a69da11f3be1b4196ff4eac134c94\": rpc error: code = NotFound desc = could not find container \"7ed1da68c1c0f52789ab0f36eb2b270c877a69da11f3be1b4196ff4eac134c94\": container with ID starting with 7ed1da68c1c0f52789ab0f36eb2b270c877a69da11f3be1b4196ff4eac134c94 not found: ID does not exist"
Sep 17 18:01:49.658390 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.658375    2576 scope.go:117] "RemoveContainer" containerID="d2adba548bca039fc010bf95833c36c7eeb76392912f348b405e5e7c3ed9eb72"
Sep 17 18:01:49.658603 ip-10-0-136-127 kubenswrapper[2576]: E0917 18:01:49.658585    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2adba548bca039fc010bf95833c36c7eeb76392912f348b405e5e7c3ed9eb72\": container with ID starting with d2adba548bca039fc010bf95833c36c7eeb76392912f348b405e5e7c3ed9eb72 not found: ID does not exist" containerID="d2adba548bca039fc010bf95833c36c7eeb76392912f348b405e5e7c3ed9eb72"
Sep 17 18:01:49.658653 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.658610    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2adba548bca039fc010bf95833c36c7eeb76392912f348b405e5e7c3ed9eb72"} err="failed to get container status \"d2adba548bca039fc010bf95833c36c7eeb76392912f348b405e5e7c3ed9eb72\": rpc error: code = NotFound desc = could not find container \"d2adba548bca039fc010bf95833c36c7eeb76392912f348b405e5e7c3ed9eb72\": container with ID starting with d2adba548bca039fc010bf95833c36c7eeb76392912f348b405e5e7c3ed9eb72 not found: ID does not exist"
Sep 17 18:01:49.672509 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.672489    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"]
Sep 17 18:01:49.681277 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:49.681257    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f845bd94d-pttm5"]
Sep 17 18:01:50.537998 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:50.537966    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52920a25-a151-4742-ba02-abc02af29a3b" path="/var/lib/kubelet/pods/52920a25-a151-4742-ba02-abc02af29a3b/volumes"
Sep 17 18:01:56.581349 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.581317    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"]
Sep 17 18:01:56.581761 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.581643    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="kserve-container"
Sep 17 18:01:56.581761 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.581656    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="kserve-container"
Sep 17 18:01:56.581761 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.581673    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kube-rbac-proxy"
Sep 17 18:01:56.581761 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.581680    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kube-rbac-proxy"
Sep 17 18:01:56.581761 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.581687    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="storage-initializer"
Sep 17 18:01:56.581761 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.581692    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="storage-initializer"
Sep 17 18:01:56.581761 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.581699    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="kube-rbac-proxy"
Sep 17 18:01:56.581761 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.581704    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="kube-rbac-proxy"
Sep 17 18:01:56.581761 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.581713    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="storage-initializer"
Sep 17 18:01:56.581761 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.581718    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="storage-initializer"
Sep 17 18:01:56.581761 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.581724    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container"
Sep 17 18:01:56.581761 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.581729    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container"
Sep 17 18:01:56.582162 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.581779    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kserve-container"
Sep 17 18:01:56.582162 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.581785    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="kube-rbac-proxy"
Sep 17 18:01:56.582162 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.581794    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ad32257-4301-4614-91e0-542b9652e6af" containerName="kube-rbac-proxy"
Sep 17 18:01:56.582162 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.581800    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="52920a25-a151-4742-ba02-abc02af29a3b" containerName="kserve-container"
Sep 17 18:01:56.584768 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.584742    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:01:56.587369 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.587298    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:01:56.587512 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.587384    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:01:56.587633 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.587617    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:01:56.587717 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.587642    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:01:56.588553 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.588523    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 18:01:56.588691 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.588533    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:01:56.588691 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.588653    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gwm88\""
Sep 17 18:01:56.596164 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.596144    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"]
Sep 17 18:01:56.685945 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.685918    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9mvm\" (UniqueName: \"kubernetes.io/projected/a968b88f-d7d1-41cf-92a0-6efb75144f26-kube-api-access-b9mvm\") pod \"isvc-canary-promote-predictor-567686cc76-nd8rj\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:01:56.685945 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.685946    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a968b88f-d7d1-41cf-92a0-6efb75144f26-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-nd8rj\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:01:56.686125 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.685985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a968b88f-d7d1-41cf-92a0-6efb75144f26-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-nd8rj\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:01:56.686125 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.686004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a968b88f-d7d1-41cf-92a0-6efb75144f26-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-nd8rj\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:01:56.686125 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.686035    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a968b88f-d7d1-41cf-92a0-6efb75144f26-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-nd8rj\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:01:56.786783 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.786756    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a968b88f-d7d1-41cf-92a0-6efb75144f26-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-nd8rj\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:01:56.786929 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.786787    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a968b88f-d7d1-41cf-92a0-6efb75144f26-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-nd8rj\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:01:56.786929 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.786810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a968b88f-d7d1-41cf-92a0-6efb75144f26-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-nd8rj\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:01:56.786929 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.786852    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9mvm\" (UniqueName: \"kubernetes.io/projected/a968b88f-d7d1-41cf-92a0-6efb75144f26-kube-api-access-b9mvm\") pod \"isvc-canary-promote-predictor-567686cc76-nd8rj\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:01:56.786929 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.786877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a968b88f-d7d1-41cf-92a0-6efb75144f26-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-nd8rj\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:01:56.787232 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.787210    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a968b88f-d7d1-41cf-92a0-6efb75144f26-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-nd8rj\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:01:56.787452 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.787434    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a968b88f-d7d1-41cf-92a0-6efb75144f26-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-nd8rj\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:01:56.787491 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.787469    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a968b88f-d7d1-41cf-92a0-6efb75144f26-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-nd8rj\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:01:56.789234 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.789217    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a968b88f-d7d1-41cf-92a0-6efb75144f26-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-nd8rj\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:01:56.796542 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.796520    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9mvm\" (UniqueName: \"kubernetes.io/projected/a968b88f-d7d1-41cf-92a0-6efb75144f26-kube-api-access-b9mvm\") pod \"isvc-canary-promote-predictor-567686cc76-nd8rj\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:01:56.896020 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:56.895993    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:01:57.177181 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:57.177159    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"]
Sep 17 18:01:57.179266 ip-10-0-136-127 kubenswrapper[2576]: W0917 18:01:57.179240    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda968b88f_d7d1_41cf_92a0_6efb75144f26.slice/crio-4747b27054155fa51e34102ffaaf58634e92b91f6cd02c1157d70bd62d382252 WatchSource:0}: Error finding container 4747b27054155fa51e34102ffaaf58634e92b91f6cd02c1157d70bd62d382252: Status 404 returned error can't find the container with id 4747b27054155fa51e34102ffaaf58634e92b91f6cd02c1157d70bd62d382252
Sep 17 18:01:57.670769 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:57.670732    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" event={"ID":"a968b88f-d7d1-41cf-92a0-6efb75144f26","Type":"ContainerStarted","Data":"e9b825d0ab33654104fb883ba5a2a97ef9a6d5b8a7937692e9a078929231c166"}
Sep 17 18:01:57.671111 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:01:57.670811    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" event={"ID":"a968b88f-d7d1-41cf-92a0-6efb75144f26","Type":"ContainerStarted","Data":"4747b27054155fa51e34102ffaaf58634e92b91f6cd02c1157d70bd62d382252"}
Sep 17 18:02:01.688439 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:02:01.688406    2576 generic.go:356] "Generic (PLEG): container finished" podID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerID="e9b825d0ab33654104fb883ba5a2a97ef9a6d5b8a7937692e9a078929231c166" exitCode=0
Sep 17 18:02:01.688818 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:02:01.688448    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" event={"ID":"a968b88f-d7d1-41cf-92a0-6efb75144f26","Type":"ContainerDied","Data":"e9b825d0ab33654104fb883ba5a2a97ef9a6d5b8a7937692e9a078929231c166"}
Sep 17 18:02:02.694111 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:02:02.694078    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" event={"ID":"a968b88f-d7d1-41cf-92a0-6efb75144f26","Type":"ContainerStarted","Data":"9c89f186d2b00ca172d6bba0d0dfa51f67e888ed48c426053e053d371764ce01"}
Sep 17 18:02:02.694423 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:02:02.694139    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" event={"ID":"a968b88f-d7d1-41cf-92a0-6efb75144f26","Type":"ContainerStarted","Data":"fb69386e26c1a7975ad46ee4ec3f5684ecd9c474e00dd5a5d60e9bf228169811"}
Sep 17 18:02:02.694423 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:02:02.694412    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:02:02.694602 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:02:02.694577    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:02:02.695809 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:02:02.695789    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:02:02.719253 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:02:02.719211    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" podStartSLOduration=2.98581204 podStartE2EDuration="6.719200476s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.733388436s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:02.717913364 +0000 UTC m=+1262.310750350" watchObservedRunningTime="2026-09-17 18:02:02.719200476 +0000 UTC m=+1262.312037458"
Sep 17 18:02:03.698645 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:02:03.698603    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:02:04.702681 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:02:04.702643    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:02:09.707849 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:02:09.707820    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:02:09.708329 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:02:09.708294    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:02:19.709033 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:02:19.708997    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:02:29.708933 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:02:29.708889    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:02:39.708889 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:02:39.708852    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:02:49.709129 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:02:49.709086    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:02:59.708779 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:02:59.708740    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:03:09.709064 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:09.709021    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:03:12.533653 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:12.533607    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:03:22.536958 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:22.536881    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:03:26.633005 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.632970    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"]
Sep 17 18:03:26.633460 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.633231    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kserve-container" containerID="cri-o://fb69386e26c1a7975ad46ee4ec3f5684ecd9c474e00dd5a5d60e9bf228169811" gracePeriod=30
Sep 17 18:03:26.633460 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.633271    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kube-rbac-proxy" containerID="cri-o://9c89f186d2b00ca172d6bba0d0dfa51f67e888ed48c426053e053d371764ce01" gracePeriod=30
Sep 17 18:03:26.722629 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.722599    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"]
Sep 17 18:03:26.725948 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.725926    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:26.728751 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.728733    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 18:03:26.738977 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.738949    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"]
Sep 17 18:03:26.839571 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.839539    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96926b59-c845-4138-90e0-9b1c4f41eafb-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:26.839713 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.839665    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96926b59-c845-4138-90e0-9b1c4f41eafb-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:26.839713 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.839695    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-984tz\" (UniqueName: \"kubernetes.io/projected/96926b59-c845-4138-90e0-9b1c4f41eafb-kube-api-access-984tz\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:26.839785 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.839717    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96926b59-c845-4138-90e0-9b1c4f41eafb-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:26.839785 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.839748    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96926b59-c845-4138-90e0-9b1c4f41eafb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:26.940492 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.940407    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96926b59-c845-4138-90e0-9b1c4f41eafb-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:26.940492 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.940443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-984tz\" (UniqueName: \"kubernetes.io/projected/96926b59-c845-4138-90e0-9b1c4f41eafb-kube-api-access-984tz\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:26.940492 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.940463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96926b59-c845-4138-90e0-9b1c4f41eafb-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:26.940492 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.940487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96926b59-c845-4138-90e0-9b1c4f41eafb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:26.940834 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.940515    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96926b59-c845-4138-90e0-9b1c4f41eafb-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:26.940892 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.940870    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96926b59-c845-4138-90e0-9b1c4f41eafb-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:26.941164 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.941145    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96926b59-c845-4138-90e0-9b1c4f41eafb-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:26.941217 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.941178    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96926b59-c845-4138-90e0-9b1c4f41eafb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:26.942927 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.942911    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96926b59-c845-4138-90e0-9b1c4f41eafb-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:26.951938 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:26.951910    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-984tz\" (UniqueName: \"kubernetes.io/projected/96926b59-c845-4138-90e0-9b1c4f41eafb-kube-api-access-984tz\") pod \"isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:27.028404 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:27.028370    2576 generic.go:356] "Generic (PLEG): container finished" podID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerID="9c89f186d2b00ca172d6bba0d0dfa51f67e888ed48c426053e053d371764ce01" exitCode=2
Sep 17 18:03:27.028556 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:27.028445    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" event={"ID":"a968b88f-d7d1-41cf-92a0-6efb75144f26","Type":"ContainerDied","Data":"9c89f186d2b00ca172d6bba0d0dfa51f67e888ed48c426053e053d371764ce01"}
Sep 17 18:03:27.035590 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:27.035553    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:27.310529 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:27.310300    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"]
Sep 17 18:03:27.313293 ip-10-0-136-127 kubenswrapper[2576]: W0917 18:03:27.313270    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96926b59_c845_4138_90e0_9b1c4f41eafb.slice/crio-e588e64ed494b087f57a4ac292233d6fe5200827d808b51255a3fcf1b8bbc4d1 WatchSource:0}: Error finding container e588e64ed494b087f57a4ac292233d6fe5200827d808b51255a3fcf1b8bbc4d1: Status 404 returned error can't find the container with id e588e64ed494b087f57a4ac292233d6fe5200827d808b51255a3fcf1b8bbc4d1
Sep 17 18:03:28.035890 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:28.035852    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" event={"ID":"96926b59-c845-4138-90e0-9b1c4f41eafb","Type":"ContainerStarted","Data":"e3ce099484743a486409e0e5fa4763ffd3afd419a42f5c28bb1983d4817842de"}
Sep 17 18:03:28.036291 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:28.035912    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" event={"ID":"96926b59-c845-4138-90e0-9b1c4f41eafb","Type":"ContainerStarted","Data":"e588e64ed494b087f57a4ac292233d6fe5200827d808b51255a3fcf1b8bbc4d1"}
Sep 17 18:03:29.703113 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:29.703075    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 18:03:31.049401 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:31.049366    2576 generic.go:356] "Generic (PLEG): container finished" podID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerID="e3ce099484743a486409e0e5fa4763ffd3afd419a42f5c28bb1983d4817842de" exitCode=0
Sep 17 18:03:31.049876 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:31.049415    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" event={"ID":"96926b59-c845-4138-90e0-9b1c4f41eafb","Type":"ContainerDied","Data":"e3ce099484743a486409e0e5fa4763ffd3afd419a42f5c28bb1983d4817842de"}
Sep 17 18:03:31.575666 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:31.575641    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:03:31.675676 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:31.675584    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a968b88f-d7d1-41cf-92a0-6efb75144f26-cabundle-cert\") pod \"a968b88f-d7d1-41cf-92a0-6efb75144f26\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") "
Sep 17 18:03:31.675676 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:31.675650    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a968b88f-d7d1-41cf-92a0-6efb75144f26-kube-rbac-proxy-sar-config\") pod \"a968b88f-d7d1-41cf-92a0-6efb75144f26\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") "
Sep 17 18:03:31.676058 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:31.675720    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a968b88f-d7d1-41cf-92a0-6efb75144f26-proxy-tls\") pod \"a968b88f-d7d1-41cf-92a0-6efb75144f26\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") "
Sep 17 18:03:31.676058 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:31.675782    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9mvm\" (UniqueName: \"kubernetes.io/projected/a968b88f-d7d1-41cf-92a0-6efb75144f26-kube-api-access-b9mvm\") pod \"a968b88f-d7d1-41cf-92a0-6efb75144f26\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") "
Sep 17 18:03:31.676058 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:31.675808    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a968b88f-d7d1-41cf-92a0-6efb75144f26-kserve-provision-location\") pod \"a968b88f-d7d1-41cf-92a0-6efb75144f26\" (UID: \"a968b88f-d7d1-41cf-92a0-6efb75144f26\") "
Sep 17 18:03:31.676058 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:31.675988    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a968b88f-d7d1-41cf-92a0-6efb75144f26-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a968b88f-d7d1-41cf-92a0-6efb75144f26" (UID: "a968b88f-d7d1-41cf-92a0-6efb75144f26"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:03:31.676058 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:31.675993    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a968b88f-d7d1-41cf-92a0-6efb75144f26-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a968b88f-d7d1-41cf-92a0-6efb75144f26" (UID: "a968b88f-d7d1-41cf-92a0-6efb75144f26"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:03:31.676245 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:31.676217    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a968b88f-d7d1-41cf-92a0-6efb75144f26-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a968b88f-d7d1-41cf-92a0-6efb75144f26" (UID: "a968b88f-d7d1-41cf-92a0-6efb75144f26"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:03:31.677921 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:31.677898    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a968b88f-d7d1-41cf-92a0-6efb75144f26-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a968b88f-d7d1-41cf-92a0-6efb75144f26" (UID: "a968b88f-d7d1-41cf-92a0-6efb75144f26"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:03:31.677973 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:31.677918    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a968b88f-d7d1-41cf-92a0-6efb75144f26-kube-api-access-b9mvm" (OuterVolumeSpecName: "kube-api-access-b9mvm") pod "a968b88f-d7d1-41cf-92a0-6efb75144f26" (UID: "a968b88f-d7d1-41cf-92a0-6efb75144f26"). InnerVolumeSpecName "kube-api-access-b9mvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:03:31.777376 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:31.777339    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a968b88f-d7d1-41cf-92a0-6efb75144f26-proxy-tls\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:03:31.777376 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:31.777371    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9mvm\" (UniqueName: \"kubernetes.io/projected/a968b88f-d7d1-41cf-92a0-6efb75144f26-kube-api-access-b9mvm\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:03:31.777376 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:31.777382    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a968b88f-d7d1-41cf-92a0-6efb75144f26-kserve-provision-location\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:03:31.777635 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:31.777391    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a968b88f-d7d1-41cf-92a0-6efb75144f26-cabundle-cert\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:03:31.777635 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:31.777400    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a968b88f-d7d1-41cf-92a0-6efb75144f26-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:03:32.056100 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.056068    2576 generic.go:356] "Generic (PLEG): container finished" podID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerID="fb69386e26c1a7975ad46ee4ec3f5684ecd9c474e00dd5a5d60e9bf228169811" exitCode=0
Sep 17 18:03:32.056469 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.056123    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"
Sep 17 18:03:32.056469 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.056150    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" event={"ID":"a968b88f-d7d1-41cf-92a0-6efb75144f26","Type":"ContainerDied","Data":"fb69386e26c1a7975ad46ee4ec3f5684ecd9c474e00dd5a5d60e9bf228169811"}
Sep 17 18:03:32.056469 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.056244    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj" event={"ID":"a968b88f-d7d1-41cf-92a0-6efb75144f26","Type":"ContainerDied","Data":"4747b27054155fa51e34102ffaaf58634e92b91f6cd02c1157d70bd62d382252"}
Sep 17 18:03:32.056469 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.056276    2576 scope.go:117] "RemoveContainer" containerID="9c89f186d2b00ca172d6bba0d0dfa51f67e888ed48c426053e053d371764ce01"
Sep 17 18:03:32.058519 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.058501    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" event={"ID":"96926b59-c845-4138-90e0-9b1c4f41eafb","Type":"ContainerStarted","Data":"36703b63c0938e3beccc50459e92d8fee1f65b69c07a8ad4e30137bae201f6af"}
Sep 17 18:03:32.058771 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.058547    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" event={"ID":"96926b59-c845-4138-90e0-9b1c4f41eafb","Type":"ContainerStarted","Data":"2c67a8c4e37c6ff13b771ef57f34b51e711d1da49235bd121fec9a4270390ee5"}
Sep 17 18:03:32.058876 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.058841    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:32.058946 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.058893    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:32.060499 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.060474    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:03:32.066774 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.066745    2576 scope.go:117] "RemoveContainer" containerID="fb69386e26c1a7975ad46ee4ec3f5684ecd9c474e00dd5a5d60e9bf228169811"
Sep 17 18:03:32.074344 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.074328    2576 scope.go:117] "RemoveContainer" containerID="e9b825d0ab33654104fb883ba5a2a97ef9a6d5b8a7937692e9a078929231c166"
Sep 17 18:03:32.081107 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.081090    2576 scope.go:117] "RemoveContainer" containerID="9c89f186d2b00ca172d6bba0d0dfa51f67e888ed48c426053e053d371764ce01"
Sep 17 18:03:32.081364 ip-10-0-136-127 kubenswrapper[2576]: E0917 18:03:32.081343    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c89f186d2b00ca172d6bba0d0dfa51f67e888ed48c426053e053d371764ce01\": container with ID starting with 9c89f186d2b00ca172d6bba0d0dfa51f67e888ed48c426053e053d371764ce01 not found: ID does not exist" containerID="9c89f186d2b00ca172d6bba0d0dfa51f67e888ed48c426053e053d371764ce01"
Sep 17 18:03:32.081434 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.081376    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c89f186d2b00ca172d6bba0d0dfa51f67e888ed48c426053e053d371764ce01"} err="failed to get container status \"9c89f186d2b00ca172d6bba0d0dfa51f67e888ed48c426053e053d371764ce01\": rpc error: code = NotFound desc = could not find container \"9c89f186d2b00ca172d6bba0d0dfa51f67e888ed48c426053e053d371764ce01\": container with ID starting with 9c89f186d2b00ca172d6bba0d0dfa51f67e888ed48c426053e053d371764ce01 not found: ID does not exist"
Sep 17 18:03:32.081434 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.081399    2576 scope.go:117] "RemoveContainer" containerID="fb69386e26c1a7975ad46ee4ec3f5684ecd9c474e00dd5a5d60e9bf228169811"
Sep 17 18:03:32.081662 ip-10-0-136-127 kubenswrapper[2576]: E0917 18:03:32.081643    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb69386e26c1a7975ad46ee4ec3f5684ecd9c474e00dd5a5d60e9bf228169811\": container with ID starting with fb69386e26c1a7975ad46ee4ec3f5684ecd9c474e00dd5a5d60e9bf228169811 not found: ID does not exist" containerID="fb69386e26c1a7975ad46ee4ec3f5684ecd9c474e00dd5a5d60e9bf228169811"
Sep 17 18:03:32.081728 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.081669    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb69386e26c1a7975ad46ee4ec3f5684ecd9c474e00dd5a5d60e9bf228169811"} err="failed to get container status \"fb69386e26c1a7975ad46ee4ec3f5684ecd9c474e00dd5a5d60e9bf228169811\": rpc error: code = NotFound desc = could not find container \"fb69386e26c1a7975ad46ee4ec3f5684ecd9c474e00dd5a5d60e9bf228169811\": container with ID starting with fb69386e26c1a7975ad46ee4ec3f5684ecd9c474e00dd5a5d60e9bf228169811 not found: ID does not exist"
Sep 17 18:03:32.081728 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.081685    2576 scope.go:117] "RemoveContainer" containerID="e9b825d0ab33654104fb883ba5a2a97ef9a6d5b8a7937692e9a078929231c166"
Sep 17 18:03:32.081921 ip-10-0-136-127 kubenswrapper[2576]: E0917 18:03:32.081903    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b825d0ab33654104fb883ba5a2a97ef9a6d5b8a7937692e9a078929231c166\": container with ID starting with e9b825d0ab33654104fb883ba5a2a97ef9a6d5b8a7937692e9a078929231c166 not found: ID does not exist" containerID="e9b825d0ab33654104fb883ba5a2a97ef9a6d5b8a7937692e9a078929231c166"
Sep 17 18:03:32.081973 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.081929    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b825d0ab33654104fb883ba5a2a97ef9a6d5b8a7937692e9a078929231c166"} err="failed to get container status \"e9b825d0ab33654104fb883ba5a2a97ef9a6d5b8a7937692e9a078929231c166\": rpc error: code = NotFound desc = could not find container \"e9b825d0ab33654104fb883ba5a2a97ef9a6d5b8a7937692e9a078929231c166\": container with ID starting with e9b825d0ab33654104fb883ba5a2a97ef9a6d5b8a7937692e9a078929231c166 not found: ID does not exist"
Sep 17 18:03:32.086342 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.086305    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" podStartSLOduration=2.493502275 podStartE2EDuration="6.086292939s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.592790664s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:32.085086188 +0000 UTC m=+1351.677923169" watchObservedRunningTime="2026-09-17 18:03:32.086292939 +0000 UTC m=+1351.679129921"
Sep 17 18:03:32.101061 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.101039    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"]
Sep 17 18:03:32.107543 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.107522    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-nd8rj"]
Sep 17 18:03:32.537880 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:32.537849    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" path="/var/lib/kubelet/pods/a968b88f-d7d1-41cf-92a0-6efb75144f26/volumes"
Sep 17 18:03:33.063348 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:33.063311    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:03:38.067804 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:38.067772    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:03:38.068268 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:38.068242    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:03:48.068429 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:48.068388    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:03:58.068980 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:03:58.068937    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:04:08.069024 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:04:08.068987    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:04:18.069183 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:04:18.069142    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:04:28.069107 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:04:28.069067    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:04:38.068242 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:04:38.068201    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:04:42.534110 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:04:42.534071    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:04:52.537839 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:04:52.537808    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:04:56.867798 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:04:56.867767    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"]
Sep 17 18:04:56.868229 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:04:56.868019    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="kserve-container" containerID="cri-o://2c67a8c4e37c6ff13b771ef57f34b51e711d1da49235bd121fec9a4270390ee5" gracePeriod=30
Sep 17 18:04:56.868229 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:04:56.868041    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="kube-rbac-proxy" containerID="cri-o://36703b63c0938e3beccc50459e92d8fee1f65b69c07a8ad4e30137bae201f6af" gracePeriod=30
Sep 17 18:04:57.415248 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:04:57.415215    2576 generic.go:356] "Generic (PLEG): container finished" podID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerID="36703b63c0938e3beccc50459e92d8fee1f65b69c07a8ad4e30137bae201f6af" exitCode=2
Sep 17 18:04:57.415428 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:04:57.415280    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" event={"ID":"96926b59-c845-4138-90e0-9b1c4f41eafb","Type":"ContainerDied","Data":"36703b63c0938e3beccc50459e92d8fee1f65b69c07a8ad4e30137bae201f6af"}
Sep 17 18:04:58.064078 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:04:58.064040    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 18:05:01.307673 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.307646    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:05:01.348156 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.348130    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96926b59-c845-4138-90e0-9b1c4f41eafb-kserve-provision-location\") pod \"96926b59-c845-4138-90e0-9b1c4f41eafb\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") "
Sep 17 18:05:01.348325 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.348187    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96926b59-c845-4138-90e0-9b1c4f41eafb-cabundle-cert\") pod \"96926b59-c845-4138-90e0-9b1c4f41eafb\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") "
Sep 17 18:05:01.348325 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.348234    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96926b59-c845-4138-90e0-9b1c4f41eafb-kube-rbac-proxy-sar-config\") pod \"96926b59-c845-4138-90e0-9b1c4f41eafb\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") "
Sep 17 18:05:01.348325 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.348282    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96926b59-c845-4138-90e0-9b1c4f41eafb-proxy-tls\") pod \"96926b59-c845-4138-90e0-9b1c4f41eafb\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") "
Sep 17 18:05:01.348325 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.348299    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-984tz\" (UniqueName: \"kubernetes.io/projected/96926b59-c845-4138-90e0-9b1c4f41eafb-kube-api-access-984tz\") pod \"96926b59-c845-4138-90e0-9b1c4f41eafb\" (UID: \"96926b59-c845-4138-90e0-9b1c4f41eafb\") "
Sep 17 18:05:01.348633 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.348484    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96926b59-c845-4138-90e0-9b1c4f41eafb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "96926b59-c845-4138-90e0-9b1c4f41eafb" (UID: "96926b59-c845-4138-90e0-9b1c4f41eafb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:05:01.348633 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.348606    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96926b59-c845-4138-90e0-9b1c4f41eafb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "96926b59-c845-4138-90e0-9b1c4f41eafb" (UID: "96926b59-c845-4138-90e0-9b1c4f41eafb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:01.348710 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.348659    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96926b59-c845-4138-90e0-9b1c4f41eafb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "96926b59-c845-4138-90e0-9b1c4f41eafb" (UID: "96926b59-c845-4138-90e0-9b1c4f41eafb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:01.350412 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.350392    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96926b59-c845-4138-90e0-9b1c4f41eafb-kube-api-access-984tz" (OuterVolumeSpecName: "kube-api-access-984tz") pod "96926b59-c845-4138-90e0-9b1c4f41eafb" (UID: "96926b59-c845-4138-90e0-9b1c4f41eafb"). InnerVolumeSpecName "kube-api-access-984tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:05:01.350468 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.350398    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96926b59-c845-4138-90e0-9b1c4f41eafb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "96926b59-c845-4138-90e0-9b1c4f41eafb" (UID: "96926b59-c845-4138-90e0-9b1c4f41eafb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:05:01.437710 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.437620    2576 generic.go:356] "Generic (PLEG): container finished" podID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerID="2c67a8c4e37c6ff13b771ef57f34b51e711d1da49235bd121fec9a4270390ee5" exitCode=0
Sep 17 18:05:01.437710 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.437683    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"
Sep 17 18:05:01.437891 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.437706    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" event={"ID":"96926b59-c845-4138-90e0-9b1c4f41eafb","Type":"ContainerDied","Data":"2c67a8c4e37c6ff13b771ef57f34b51e711d1da49235bd121fec9a4270390ee5"}
Sep 17 18:05:01.437891 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.437770    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp" event={"ID":"96926b59-c845-4138-90e0-9b1c4f41eafb","Type":"ContainerDied","Data":"e588e64ed494b087f57a4ac292233d6fe5200827d808b51255a3fcf1b8bbc4d1"}
Sep 17 18:05:01.437891 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.437797    2576 scope.go:117] "RemoveContainer" containerID="36703b63c0938e3beccc50459e92d8fee1f65b69c07a8ad4e30137bae201f6af"
Sep 17 18:05:01.446136 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.446118    2576 scope.go:117] "RemoveContainer" containerID="2c67a8c4e37c6ff13b771ef57f34b51e711d1da49235bd121fec9a4270390ee5"
Sep 17 18:05:01.449398 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.449381    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96926b59-c845-4138-90e0-9b1c4f41eafb-cabundle-cert\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:05:01.449500 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.449400    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96926b59-c845-4138-90e0-9b1c4f41eafb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:05:01.449500 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.449414    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96926b59-c845-4138-90e0-9b1c4f41eafb-proxy-tls\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:05:01.449500 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.449425    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-984tz\" (UniqueName: \"kubernetes.io/projected/96926b59-c845-4138-90e0-9b1c4f41eafb-kube-api-access-984tz\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:05:01.449500 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.449433    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96926b59-c845-4138-90e0-9b1c4f41eafb-kserve-provision-location\") on node \"ip-10-0-136-127.ec2.internal\" DevicePath \"\""
Sep 17 18:05:01.453112 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.453096    2576 scope.go:117] "RemoveContainer" containerID="e3ce099484743a486409e0e5fa4763ffd3afd419a42f5c28bb1983d4817842de"
Sep 17 18:05:01.460149 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.460071    2576 scope.go:117] "RemoveContainer" containerID="36703b63c0938e3beccc50459e92d8fee1f65b69c07a8ad4e30137bae201f6af"
Sep 17 18:05:01.460367 ip-10-0-136-127 kubenswrapper[2576]: E0917 18:05:01.460348    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36703b63c0938e3beccc50459e92d8fee1f65b69c07a8ad4e30137bae201f6af\": container with ID starting with 36703b63c0938e3beccc50459e92d8fee1f65b69c07a8ad4e30137bae201f6af not found: ID does not exist" containerID="36703b63c0938e3beccc50459e92d8fee1f65b69c07a8ad4e30137bae201f6af"
Sep 17 18:05:01.460431 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.460386    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36703b63c0938e3beccc50459e92d8fee1f65b69c07a8ad4e30137bae201f6af"} err="failed to get container status \"36703b63c0938e3beccc50459e92d8fee1f65b69c07a8ad4e30137bae201f6af\": rpc error: code = NotFound desc = could not find container \"36703b63c0938e3beccc50459e92d8fee1f65b69c07a8ad4e30137bae201f6af\": container with ID starting with 36703b63c0938e3beccc50459e92d8fee1f65b69c07a8ad4e30137bae201f6af not found: ID does not exist"
Sep 17 18:05:01.460431 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.460404    2576 scope.go:117] "RemoveContainer" containerID="2c67a8c4e37c6ff13b771ef57f34b51e711d1da49235bd121fec9a4270390ee5"
Sep 17 18:05:01.460710 ip-10-0-136-127 kubenswrapper[2576]: E0917 18:05:01.460688    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c67a8c4e37c6ff13b771ef57f34b51e711d1da49235bd121fec9a4270390ee5\": container with ID starting with 2c67a8c4e37c6ff13b771ef57f34b51e711d1da49235bd121fec9a4270390ee5 not found: ID does not exist" containerID="2c67a8c4e37c6ff13b771ef57f34b51e711d1da49235bd121fec9a4270390ee5"
Sep 17 18:05:01.460766 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.460719    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c67a8c4e37c6ff13b771ef57f34b51e711d1da49235bd121fec9a4270390ee5"} err="failed to get container status \"2c67a8c4e37c6ff13b771ef57f34b51e711d1da49235bd121fec9a4270390ee5\": rpc error: code = NotFound desc = could not find container \"2c67a8c4e37c6ff13b771ef57f34b51e711d1da49235bd121fec9a4270390ee5\": container with ID starting with 2c67a8c4e37c6ff13b771ef57f34b51e711d1da49235bd121fec9a4270390ee5 not found: ID does not exist"
Sep 17 18:05:01.460766 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.460740    2576 scope.go:117] "RemoveContainer" containerID="e3ce099484743a486409e0e5fa4763ffd3afd419a42f5c28bb1983d4817842de"
Sep 17 18:05:01.461025 ip-10-0-136-127 kubenswrapper[2576]: E0917 18:05:01.461009    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3ce099484743a486409e0e5fa4763ffd3afd419a42f5c28bb1983d4817842de\": container with ID starting with e3ce099484743a486409e0e5fa4763ffd3afd419a42f5c28bb1983d4817842de not found: ID does not exist" containerID="e3ce099484743a486409e0e5fa4763ffd3afd419a42f5c28bb1983d4817842de"
Sep 17 18:05:01.461079 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.461028    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ce099484743a486409e0e5fa4763ffd3afd419a42f5c28bb1983d4817842de"} err="failed to get container status \"e3ce099484743a486409e0e5fa4763ffd3afd419a42f5c28bb1983d4817842de\": rpc error: code = NotFound desc = could not find container \"e3ce099484743a486409e0e5fa4763ffd3afd419a42f5c28bb1983d4817842de\": container with ID starting with e3ce099484743a486409e0e5fa4763ffd3afd419a42f5c28bb1983d4817842de not found: ID does not exist"
Sep 17 18:05:01.461429 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.461413    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"]
Sep 17 18:05:01.464631 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:01.464610    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6ccf956dbc-pqrsp"]
Sep 17 18:05:02.536921 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:05:02.536885    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" path="/var/lib/kubelet/pods/96926b59-c845-4138-90e0-9b1c4f41eafb/volumes"
Sep 17 18:06:02.523356 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:06:02.523329    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/ovn-acl-logging/0.log"
Sep 17 18:06:02.525645 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:06:02.525617    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/ovn-acl-logging/0.log"
Sep 17 18:07:31.599706 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:31.599663    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7cdxt_a540f631-c6e3-4f4f-b9a5-4fbeb92f41e2/global-pull-secret-syncer/0.log"
Sep 17 18:07:31.730961 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:31.730924    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mb5dq_feff5f74-e1df-4fdc-b2f3-ebd27908a886/konnectivity-agent/0.log"
Sep 17 18:07:31.819751 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:31.819721    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-127.ec2.internal_dd6c8d1b0dcecbce259dac62f1441a28/haproxy/0.log"
Sep 17 18:07:36.555041 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:36.555004    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-vjfbj_ed16268e-bcee-4fe4-bf62-3761059fdd9c/monitoring-plugin/0.log"
Sep 17 18:07:36.787836 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:36.787805    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qfb65_61fde122-4e11-434e-ba57-0cd18b35d076/node-exporter/0.log"
Sep 17 18:07:36.815361 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:36.815241    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qfb65_61fde122-4e11-434e-ba57-0cd18b35d076/kube-rbac-proxy/0.log"
Sep 17 18:07:36.839704 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:36.839675    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qfb65_61fde122-4e11-434e-ba57-0cd18b35d076/init-textfile/0.log"
Sep 17 18:07:36.877685 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:36.877647    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-pdrc6_c61fd583-592e-4fee-b0fa-cc18e365b9ab/kube-rbac-proxy-main/0.log"
Sep 17 18:07:36.907047 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:36.907023    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-pdrc6_c61fd583-592e-4fee-b0fa-cc18e365b9ab/kube-rbac-proxy-self/0.log"
Sep 17 18:07:36.938861 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:36.938833    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-pdrc6_c61fd583-592e-4fee-b0fa-cc18e365b9ab/openshift-state-metrics/0.log"
Sep 17 18:07:37.289961 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:37.289932    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-zs2sz_fa773c89-0150-46a3-b187-08d5a995b39f/prometheus-operator-admission-webhook/0.log"
Sep 17 18:07:37.460409 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:37.460377    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f8965fb5-79sgc_cd399548-098b-4070-9532-d8730d35d4d3/thanos-query/0.log"
Sep 17 18:07:37.492711 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:37.492661    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f8965fb5-79sgc_cd399548-098b-4070-9532-d8730d35d4d3/kube-rbac-proxy-web/0.log"
Sep 17 18:07:37.517339 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:37.517304    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f8965fb5-79sgc_cd399548-098b-4070-9532-d8730d35d4d3/kube-rbac-proxy/0.log"
Sep 17 18:07:37.546626 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:37.546529    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f8965fb5-79sgc_cd399548-098b-4070-9532-d8730d35d4d3/prom-label-proxy/0.log"
Sep 17 18:07:37.576635 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:37.576612    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f8965fb5-79sgc_cd399548-098b-4070-9532-d8730d35d4d3/kube-rbac-proxy-rules/0.log"
Sep 17 18:07:37.609115 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:37.609093    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f8965fb5-79sgc_cd399548-098b-4070-9532-d8730d35d4d3/kube-rbac-proxy-metrics/0.log"
Sep 17 18:07:39.536465 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.536426    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"]
Sep 17 18:07:39.536874 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.536739    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="kserve-container"
Sep 17 18:07:39.536874 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.536750    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="kserve-container"
Sep 17 18:07:39.536874 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.536760    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="storage-initializer"
Sep 17 18:07:39.536874 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.536765    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="storage-initializer"
Sep 17 18:07:39.536874 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.536774    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kserve-container"
Sep 17 18:07:39.536874 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.536780    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kserve-container"
Sep 17 18:07:39.536874 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.536789    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="kube-rbac-proxy"
Sep 17 18:07:39.536874 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.536794    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="kube-rbac-proxy"
Sep 17 18:07:39.536874 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.536803    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="storage-initializer"
Sep 17 18:07:39.536874 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.536808    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="storage-initializer"
Sep 17 18:07:39.536874 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.536818    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kube-rbac-proxy"
Sep 17 18:07:39.536874 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.536823    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kube-rbac-proxy"
Sep 17 18:07:39.536874 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.536866    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kserve-container"
Sep 17 18:07:39.536874 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.536875    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a968b88f-d7d1-41cf-92a0-6efb75144f26" containerName="kube-rbac-proxy"
Sep 17 18:07:39.536874 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.536882    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="kube-rbac-proxy"
Sep 17 18:07:39.537323 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.536889    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="96926b59-c845-4138-90e0-9b1c4f41eafb" containerName="kserve-container"
Sep 17 18:07:39.540012 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.539996    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"
Sep 17 18:07:39.545313 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.545290    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-66kd5\"/\"kube-root-ca.crt\""
Sep 17 18:07:39.545692 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.545666    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-66kd5\"/\"openshift-service-ca.crt\""
Sep 17 18:07:39.546484 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.546458    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-66kd5\"/\"default-dockercfg-frhg2\""
Sep 17 18:07:39.559936 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.559905    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"]
Sep 17 18:07:39.646419 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.646383    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nx89\" (UniqueName: \"kubernetes.io/projected/981b3d89-cc6d-4a09-b573-7454bf9d4ead-kube-api-access-2nx89\") pod \"perf-node-gather-daemonset-zsjvx\" (UID: \"981b3d89-cc6d-4a09-b573-7454bf9d4ead\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"
Sep 17 18:07:39.646419 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.646425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/981b3d89-cc6d-4a09-b573-7454bf9d4ead-sys\") pod \"perf-node-gather-daemonset-zsjvx\" (UID: \"981b3d89-cc6d-4a09-b573-7454bf9d4ead\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"
Sep 17 18:07:39.646665 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.646454    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/981b3d89-cc6d-4a09-b573-7454bf9d4ead-podres\") pod \"perf-node-gather-daemonset-zsjvx\" (UID: \"981b3d89-cc6d-4a09-b573-7454bf9d4ead\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"
Sep 17 18:07:39.646665 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.646527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/981b3d89-cc6d-4a09-b573-7454bf9d4ead-lib-modules\") pod \"perf-node-gather-daemonset-zsjvx\" (UID: \"981b3d89-cc6d-4a09-b573-7454bf9d4ead\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"
Sep 17 18:07:39.646665 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.646585    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/981b3d89-cc6d-4a09-b573-7454bf9d4ead-proc\") pod \"perf-node-gather-daemonset-zsjvx\" (UID: \"981b3d89-cc6d-4a09-b573-7454bf9d4ead\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"
Sep 17 18:07:39.747958 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.747921    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/981b3d89-cc6d-4a09-b573-7454bf9d4ead-lib-modules\") pod \"perf-node-gather-daemonset-zsjvx\" (UID: \"981b3d89-cc6d-4a09-b573-7454bf9d4ead\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"
Sep 17 18:07:39.747958 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.747959    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/981b3d89-cc6d-4a09-b573-7454bf9d4ead-proc\") pod \"perf-node-gather-daemonset-zsjvx\" (UID: \"981b3d89-cc6d-4a09-b573-7454bf9d4ead\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"
Sep 17 18:07:39.748204 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.747997    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nx89\" (UniqueName: \"kubernetes.io/projected/981b3d89-cc6d-4a09-b573-7454bf9d4ead-kube-api-access-2nx89\") pod \"perf-node-gather-daemonset-zsjvx\" (UID: \"981b3d89-cc6d-4a09-b573-7454bf9d4ead\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"
Sep 17 18:07:39.748204 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.748016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/981b3d89-cc6d-4a09-b573-7454bf9d4ead-sys\") pod \"perf-node-gather-daemonset-zsjvx\" (UID: \"981b3d89-cc6d-4a09-b573-7454bf9d4ead\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"
Sep 17 18:07:39.748204 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.748045    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/981b3d89-cc6d-4a09-b573-7454bf9d4ead-podres\") pod \"perf-node-gather-daemonset-zsjvx\" (UID: \"981b3d89-cc6d-4a09-b573-7454bf9d4ead\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"
Sep 17 18:07:39.748204 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.748119    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/981b3d89-cc6d-4a09-b573-7454bf9d4ead-sys\") pod \"perf-node-gather-daemonset-zsjvx\" (UID: \"981b3d89-cc6d-4a09-b573-7454bf9d4ead\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"
Sep 17 18:07:39.748204 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.748121    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/981b3d89-cc6d-4a09-b573-7454bf9d4ead-proc\") pod \"perf-node-gather-daemonset-zsjvx\" (UID: \"981b3d89-cc6d-4a09-b573-7454bf9d4ead\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"
Sep 17 18:07:39.748204 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.748121    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/981b3d89-cc6d-4a09-b573-7454bf9d4ead-lib-modules\") pod \"perf-node-gather-daemonset-zsjvx\" (UID: \"981b3d89-cc6d-4a09-b573-7454bf9d4ead\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"
Sep 17 18:07:39.748204 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.748186    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/981b3d89-cc6d-4a09-b573-7454bf9d4ead-podres\") pod \"perf-node-gather-daemonset-zsjvx\" (UID: \"981b3d89-cc6d-4a09-b573-7454bf9d4ead\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"
Sep 17 18:07:39.757392 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.757362    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nx89\" (UniqueName: \"kubernetes.io/projected/981b3d89-cc6d-4a09-b573-7454bf9d4ead-kube-api-access-2nx89\") pod \"perf-node-gather-daemonset-zsjvx\" (UID: \"981b3d89-cc6d-4a09-b573-7454bf9d4ead\") " pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"
Sep 17 18:07:39.849456 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:39.849418    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"
Sep 17 18:07:40.124639 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:40.124545    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"]
Sep 17 18:07:40.128758 ip-10-0-136-127 kubenswrapper[2576]: W0917 18:07:40.128724    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod981b3d89_cc6d_4a09_b573_7454bf9d4ead.slice/crio-e74fe9b913faa7e668c2749ae2cbcc6fe8d0516affe148ee63a7db3bdcce2b14 WatchSource:0}: Error finding container e74fe9b913faa7e668c2749ae2cbcc6fe8d0516affe148ee63a7db3bdcce2b14: Status 404 returned error can't find the container with id e74fe9b913faa7e668c2749ae2cbcc6fe8d0516affe148ee63a7db3bdcce2b14
Sep 17 18:07:41.046441 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:41.046406    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx" event={"ID":"981b3d89-cc6d-4a09-b573-7454bf9d4ead","Type":"ContainerStarted","Data":"34b32ba529a524e6ed02052dc35bc047f06855eb0417617cee4bb105f36000e8"}
Sep 17 18:07:41.046967 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:41.046470    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx" event={"ID":"981b3d89-cc6d-4a09-b573-7454bf9d4ead","Type":"ContainerStarted","Data":"e74fe9b913faa7e668c2749ae2cbcc6fe8d0516affe148ee63a7db3bdcce2b14"}
Sep 17 18:07:41.046967 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:41.046499    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"
Sep 17 18:07:41.063542 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:41.063469    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx" podStartSLOduration=2.063451931 podStartE2EDuration="2.063451931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:41.063043 +0000 UTC m=+1600.655879983" watchObservedRunningTime="2026-09-17 18:07:41.063451931 +0000 UTC m=+1600.656288914"
Sep 17 18:07:41.473251 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:41.473225    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qjlh9_f80b68ad-44e6-40fd-828b-aa73ab059235/dns/0.log"
Sep 17 18:07:41.498846 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:41.498818    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qjlh9_f80b68ad-44e6-40fd-828b-aa73ab059235/kube-rbac-proxy/0.log"
Sep 17 18:07:41.688455 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:41.688421    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vmxsb_d54d9137-d225-4657-b601-b1a8bdee3b68/dns-node-resolver/0.log"
Sep 17 18:07:42.159953 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:42.159922    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7rmhq_0f0fc3cd-4a4b-4fd3-ad2b-aa9865f74852/node-ca/0.log"
Sep 17 18:07:43.394797 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:43.394765    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5zhfj_2353bb3f-e217-4fb7-9202-8c567d93cf29/serve-healthcheck-canary/0.log"
Sep 17 18:07:44.087234 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:44.087202    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xrbg5_1c6fd988-98c7-4e47-b583-64625bdb7cb2/kube-rbac-proxy/0.log"
Sep 17 18:07:44.113881 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:44.113853    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xrbg5_1c6fd988-98c7-4e47-b583-64625bdb7cb2/exporter/0.log"
Sep 17 18:07:44.138994 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:44.138964    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xrbg5_1c6fd988-98c7-4e47-b583-64625bdb7cb2/extractor/0.log"
Sep 17 18:07:46.473532 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:46.473499    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-5c5cdf4877-hh4ws_8226ce9a-a40d-4986-baf4-bd198131b36e/manager/0.log"
Sep 17 18:07:46.506506 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:46.506470    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-26vpf_5457555d-ad07-4268-b61e-00d85a39d35a/manager/0.log"
Sep 17 18:07:46.528305 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:46.528279    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-9lh52_d6970b2e-d104-4dbb-837e-e1c286d9fd77/server/0.log"
Sep 17 18:07:46.653328 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:46.653294    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-qqbzm_37c415ed-9fcf-4895-b22f-d87508b4f691/manager/0.log"
Sep 17 18:07:47.061482 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:47.061451    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-66kd5/perf-node-gather-daemonset-zsjvx"
Sep 17 18:07:52.962159 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:52.962131    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7qls8_0623a097-d7b1-4be4-be54-a29b81cc540a/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:07:52.987552 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:52.987525    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7qls8_0623a097-d7b1-4be4-be54-a29b81cc540a/egress-router-binary-copy/0.log"
Sep 17 18:07:53.014035 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:53.014010    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7qls8_0623a097-d7b1-4be4-be54-a29b81cc540a/cni-plugins/0.log"
Sep 17 18:07:53.038509 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:53.038485    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7qls8_0623a097-d7b1-4be4-be54-a29b81cc540a/bond-cni-plugin/0.log"
Sep 17 18:07:53.064054 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:53.064014    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7qls8_0623a097-d7b1-4be4-be54-a29b81cc540a/routeoverride-cni/0.log"
Sep 17 18:07:53.087366 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:53.087341    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7qls8_0623a097-d7b1-4be4-be54-a29b81cc540a/whereabouts-cni-bincopy/0.log"
Sep 17 18:07:53.115202 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:53.115177    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7qls8_0623a097-d7b1-4be4-be54-a29b81cc540a/whereabouts-cni/0.log"
Sep 17 18:07:53.519046 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:53.519017    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rdmsd_28657902-dde3-4d9b-bbaa-ce6abfefc4e9/kube-multus/0.log"
Sep 17 18:07:53.782379 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:53.782309    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-z44z2_84810a50-2551-4d78-ac6d-4d17f1524389/network-metrics-daemon/0.log"
Sep 17 18:07:53.807531 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:53.807506    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-z44z2_84810a50-2551-4d78-ac6d-4d17f1524389/kube-rbac-proxy/0.log"
Sep 17 18:07:55.107579 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:55.107532    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/ovn-controller/0.log"
Sep 17 18:07:55.130522 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:55.130498    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/ovn-acl-logging/1.log"
Sep 17 18:07:55.130697 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:55.130533    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/ovn-acl-logging/0.log"
Sep 17 18:07:55.150612 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:55.150588    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/kube-rbac-proxy-node/0.log"
Sep 17 18:07:55.182253 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:55.182205    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:07:55.209953 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:55.209929    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/northd/0.log"
Sep 17 18:07:55.240292 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:55.240265    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/nbdb/0.log"
Sep 17 18:07:55.266540 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:55.266515    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/sbdb/0.log"
Sep 17 18:07:55.306159 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:55.306127    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4nxf_5bc15f81-98ec-416f-a1bd-9f5d6d2a367b/ovnkube-controller/0.log"
Sep 17 18:07:56.422789 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:56.422762    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-qrrjr_365faef8-3867-42a8-a227-d81293771f14/network-check-target-container/0.log"
Sep 17 18:07:57.615685 ip-10-0-136-127 kubenswrapper[2576]: I0917 18:07:57.615656    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-mh9jw_8bacc5d4-bc80-46b0-93f5-c6dd572df3c1/iptables-alerter/0.log"